Any successful story of debugging linux 4.13 with qemu 2.10 and gdb 8.01?

Any successful story of debugging linux 4.13 with qemu 2.10 and gdb 8.01?

[jjDaNiMoTh]

Hello all,

As the title says, any of you have successfully tried to debug Linux
4.13 with QEMU? My problem is that it is not possible to break (even
with hbreak) in any function of the kernel, from the most used
(schedule or spin_lock) to the most obvious (uptime_proc_show when
/proc/uptime is read).

To be more specific, QEMU correctly stops if I put "-s -S" on the
command line, and perfectly continue the execution of the kernel when
I connect gdb using 'target remote :1234'. However, any breakpoint is
simply ignored.

I have followed any possible tutorial and any possible answer to
similar user questions. I also tried to do the same on a 16.04 version
of Ubuntu, but the result is always the same. In particular, I have
followed https://stackoverflow.com/questions/11408041/how-to-
debug-the-linux-kernel-with-gdb-and-qemu,
using both a busybox image and a raw filesystem image of Archlinux.

There is any known problem in debugging Linux? How can I "debug" this
debug process?

Thank you!

-- 
Stay open, be free.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20170920/ff079837/attachment.html 

Any successful story of debugging linux 4.13 with qemu 2.10 and gdb 8.01?

[慕冬亮]

2017-09-20 4:18 GMT-04:00 jjDaNiMoTh <jjdanimoth@gmail.com>:
> Hello all,
>
> As the title says, any of you have successfully tried to debug Linux
> 4.13 with QEMU? My problem is that it is not possible to break (even
> with hbreak) in any function of the kernel, from the most used
> (schedule or spin_lock) to the most obvious (uptime_proc_show when
> /proc/uptime is read).
>
> To be more specific, QEMU correctly stops if I put "-s -S" on the
> command line, and perfectly continue the execution of the kernel when
> I connect gdb using 'target remote :1234'. However, any breakpoint is
> simply ignored.
>
I encountered this problem in my debian testing. Any "break" or
"hbreak" point is not triggered
even if I set breakpoint at "start_kernel".

The version of those software are as follows :
qemu                                  1:2.8+dfsg-7                   amd64
gdb                                   7.12-6                               amd64
kernel                                  4.13.0 (build by myself)

But I found one interesting phenomenon:

If you try to "Ctrl + C" to stop the gdb when you see busybox is
already working, you will see one special error:

Remote 'g' packet reply is too long:
000000000000000017d11000008ef4810120008000000000fdfb8b07000000000d352828000000004040010000000000903fe081ffffffff883fe081ffffffff00000000000e0000ffffffffffe0ffffffffffff07ffffffffffffffff9fffff17d11000008ef4810000000000800000fffffffff8ffffffffff0000ffffffff2ddbf481ffffffff4600000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007f0300000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000801f0000

I try to use busybox and one simple "helloworld" rootfs. The result
shows it is not related with rootfs.
It seems gdb does not handle register length of amd64 due to
https://stackoverflow.com/questions/11408041/how-to-debug-the-linux-kernel-with-gdb-and-qemu.
But until now, I did not figure out the reason why this error occurs
and how to fix it.


--
My best regards to you.

     No System Is Safe!
     Dongliang Mu

> I have followed any possible tutorial and any possible answer to
> similar user questions. I also tried to do the same on a 16.04 version
> of Ubuntu, but the result is always the same. In particular, I have
> followed
> https://stackoverflow.com/questions/11408041/how-to-debug-the-linux-kernel-with-gdb-and-qemu,
> using both a busybox image and a raw filesystem image of Archlinux.
>

> There is any known problem in debugging Linux? How can I "debug" this
> debug process?
>
> Thank you!
>
> --
> Stay open, be free.
>
> _______________________________________________
> Kernelnewbies mailing list
> Kernelnewbies at kernelnewbies.org
> https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
>

Any successful story of debugging linux 4.13 with qemu 2.10 and gdb 8.01?

[jjDaNiMoTh]

On 20/09/17 at 11:20am, ??? wrote:
> 2017-09-20 4:18 GMT-04:00 jjDaNiMoTh <jjdanimoth@gmail.com>:
> > Hello all,
> >
> > As the title says, any of you have successfully tried to debug Linux
> > 4.13 with QEMU?

> I encountered this problem in my debian testing. Any "break" or
> "hbreak" point is not triggered
> even if I set breakpoint at "start_kernel".
[cut]

So, after two weeks of a headache I took the courage to send an email
here, and after only 3 hours after the email (!!!) I found the cause of
the problem.

Without being too long, the problem is KALSR. You have to disable it
disabling the kernel option "Randomize the address of the kernel image (KASLR)",
inside Processor type and features. In this way, gdb can translate
all the symbols correctly (symbols <-> addresses).

I really don't know how the developers test their code... because there
is no evidence for this.

> But I found one interesting phenomenon:
> 
> If you try to "Ctrl + C" to stop the gdb when you see busybox is
> already working, you will see one special error:
> 
> Remote 'g' packet reply is too long:
[cut]

The answer to this issue is found easily on the internet. You have to
stop using -S, because it creates problems. Please take a look to [1].


Thanks for your reply and have a nice day!

JJD

[1] https://www.google.com/search?q=packet+reply+too+long+gdb+linux

Any successful story of debugging linux 4.13 with qemu 2.10 and gdb 8.01?

[Greg KH]

On Wed, Sep 20, 2017 at 09:29:17PM +0200, jjDaNiMoTh wrote:
> On 20/09/17 at 11:20am, ??? wrote:
> > 2017-09-20 4:18 GMT-04:00 jjDaNiMoTh <jjdanimoth@gmail.com>:
> > > Hello all,
> > >
> > > As the title says, any of you have successfully tried to debug Linux
> > > 4.13 with QEMU?
> 
> > I encountered this problem in my debian testing. Any "break" or
> > "hbreak" point is not triggered
> > even if I set breakpoint at "start_kernel".
> [cut]
> 
> So, after two weeks of a headache I took the courage to send an email
> here, and after only 3 hours after the email (!!!) I found the cause of
> the problem.
> 
> Without being too long, the problem is KALSR. You have to disable it
> disabling the kernel option "Randomize the address of the kernel image (KASLR)",
> inside Processor type and features. In this way, gdb can translate
> all the symbols correctly (symbols <-> addresses).
> 
> I really don't know how the developers test their code... because there
> is no evidence for this.

Why do you think that it requires developers to use a debugger and qemu
to test their code?  I've never used a debugger on the kernel in all of
my years of kernel development...

Glad you found a solution and posted it.

greg k-h

Any successful story of debugging linux 4.13 with qemu 2.10 and gdb 8.01?

[Saket Sinha]

Hi,

>> Without being too long, the problem is KALSR. You have to disable it
>> disabling the kernel option "Randomize the address of the kernel image (KASLR)",
>> inside Processor type and features. In this way, gdb can translate
>> all the symbols correctly (symbols <-> addresses).
>>
>> I really don't know how the developers test their code... because there
>> is no evidence for this.
>
> Why do you think that it requires developers to use a debugger and qemu
> to test their code?  I've never used a debugger on the kernel in all of
> my years of kernel development...
>

I agree the developers should not be doing it.

It is documented that gdb/kgdb will not work if KALSAR is enabled.

Please refer -
https://github.com/torvalds/linux/commit/e604f1cb85367d2e5fd4cf253296d190996da81a


Regards,
Saket Sinha

Any successful story of debugging linux 4.13 with qemu 2.10 and gdb 8.01?

[jjDaNiMoTh]

2017-09-21 9:21 GMT+02:00 Greg KH <greg@kroah.com>:

>
> Why do you think that it requires developers to use a debugger and qemu
> to test their code?  I've never used a debugger on the kernel in all of
> my years of kernel development...
>
>
I imagined that, at least in early development phases, developers can
benefit
from the use of a debugger, but it seems that I was wrong. Of course, for
newbies like me, a debugger is essential to understand flow and relations
between kernel code, and I'm happy that this solution is already in the
documentation (unfortunately, not for the version I was using, 4.13). Maybe
the
patch pointed out by Saket can be backported to releases which have the
possibility to enable KALSR, such as 4.9 (LTS) and 4.13 itself (stable)?

[1]
https://www.kernel.org/doc/html/v4.13/dev-tools/gdb-kernel-debugging.html

JJD


-- 
Stay open, be free.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20170921/e139510f/attachment-0001.html 

Any successful story of debugging linux 4.13 with qemu 2.10 and gdb 8.01?

[Greg KH]

On Thu, Sep 21, 2017 at 10:43:08AM +0200, jjDaNiMoTh wrote:
> 
> 
> 2017-09-21 9:21 GMT+02:00 Greg KH <greg@kroah.com>:
> 
>    
>     Why do you think that it requires developers to use a debugger and qemu
>     to test their code?? I've never used a debugger on the kernel in all of
>     my years of kernel development...
> 
> 
> 
> I imagined that, at least in early development phases, developers can benefit
> from the use of a debugger, but it seems that I was wrong. Of course, for
> newbies like me, a debugger is essential to understand flow and relations
> between kernel code, and I'm happy that this solution is already in the
> documentation (unfortunately, not for the version I was using, 4.13). Maybe the
> patch pointed out by Saket can be backported to releases which have the
> possibility to enable KALSR, such as 4.9 (LTS) and 4.13 itself (stable)?

Sure, you can request it, look at the documentation here for how to do
that:

    https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html

Any successful story of debugging linux 4.13 with qemu 2.10 and gdb 8.01?

[慕冬亮]

2017-09-20 15:29 GMT-04:00 jjDaNiMoTh <jjdanimoth@gmail.com>:
> On 20/09/17 at 11:20am, ??? wrote:
>> 2017-09-20 4:18 GMT-04:00 jjDaNiMoTh <jjdanimoth@gmail.com>:
>> > Hello all,
>> >
>> > As the title says, any of you have successfully tried to debug Linux
>> > 4.13 with QEMU?
>
>> I encountered this problem in my debian testing. Any "break" or
>> "hbreak" point is not triggered
>> even if I set breakpoint at "start_kernel".
> [cut]
>
> So, after two weeks of a headache I took the courage to send an email
> here, and after only 3 hours after the email (!!!) I found the cause of
> the problem.
>
> Without being too long, the problem is KALSR. You have to disable it
> disabling the kernel option "Randomize the address of the kernel image (KASLR)",
> inside Processor type and features. In this way, gdb can translate
> all the symbols correctly (symbols <-> addresses).
>
> I really don't know how the developers test their code... because there
> is no evidence for this.
>
>> But I found one interesting phenomenon:
>>
>> If you try to "Ctrl + C" to stop the gdb when you see busybox is
>> already working, you will see one special error:
>>
>> Remote 'g' packet reply is too long:
> [cut]
>
> The answer to this issue is found easily on the internet. You have to
> stop using -S, because it creates problems. Please take a look to [1].

I have seen this solution during searching answers. But without "-S"
(-S stalls QEMU CPU at startup), the kernel will not stop util it reaches
busybox command line.
In other word, you have no method to debug the Linux early boot process.

So I try to find solution to fix it and debug the kernel from
start_kernel to review
the early boot process of linux Kernel.

I do not know how one senior developer debugs kernel code. I just want to
use qemu, gdb to debug and learn how kernel works.

>
>
> Thanks for your reply and have a nice day!
>
> JJD
>
> [1] https://www.google.com/search?q=packet+reply+too+long+gdb+linux

Any successful story of debugging linux 4.13 with qemu 2.10 and gdb 8.01?

[Saket Sinha]

Hi,

>
> I have seen this solution during searching answers. But without "-S"
> (-S stalls QEMU CPU at startup), the kernel will not stop util it reaches
> busybox command line.
> In other word, you have no method to debug the Linux early boot process.
>


The early boot process is highly architecture dependent.
You can emulate a virtual device close to what you have as a real
device and write drivers for that.

Please follow the below link as to how Android emulator(which uses
qemu at backend) does it -
https://android.googlesource.com/platform/external/qemu/+/master/docs/GOLDFISH-VIRTUAL-HARDWARE.TXT


Regards,
Saket Sinha