Ftrace, KASLR and gdb

Ftrace, KASLR and gdb

[alexandre.ferrieux]

Hi,

Ftrace is a jewel to dig into the kernel, be it for troubleshooting, perf tuning 
or just understanding.
But when one needs to disassemble the running kernel (eg to move kprobes around 
in a function, in order to understand a given code path), KASLR makes it 
impossible for gdb to get useful symbol addresses, even with a debug image.
That said, /proc/kallsyms always gives the accurate, present symbol addresses. 
But, to my knowledge, gdb isn't able to import /proc/kallsyms as a symbol table.
To circumvent this, I've written a small userland tool, usling libbfd, that 
creates an ELF file out of /proc/kallsyms. Passing this ELF file to gdb instead 
of "vmlinux", and /proc/kcore as core, then allows for a perfect gdb session on 
the running kernel. Of course this ELF file is only valid until the next reboot, 
but that's okay as its creation is fast.

Now, my question: did I miss an alternative ?

In other words, is there some kind of "kallsyms plug-in" for gdb somewhere ?
Or, taking the problem from the other side, some kernel module exposing a 
"/proc/kallsyms.elf" or similar, for direct consumption by gdb ?
Or another method, that people routinely use for the same purpose ?

Thanks in advance,

-Alex




____________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

Re: Ftrace, KASLR and gdb

[Steven Rostedt]

On Thu, 9 May 2024 14:44:22 +0200
alexandre.ferrieux@orange.com wrote:

> Hi,
> 
> Ftrace is a jewel to dig into the kernel, be it for troubleshooting, perf tuning 
> or just understanding.
> But when one needs to disassemble the running kernel (eg to move kprobes around 
> in a function, in order to understand a given code path), KASLR makes it 
> impossible for gdb to get useful symbol addresses, even with a debug image.

Really? Can't you just use a function name plus offset? For instance, I
do this all the time:

(gdb) li vfs_write+0xc3
0xffffffff812e2513 is in vfs_write (/work/git/linux-trace.git/fs/read_write.c:592).
587                     return ret;
588             if (count > MAX_RW_COUNT)
589                     count =  MAX_RW_COUNT;
590             file_start_write(file);
591             if (file->f_op->write)
592                     ret = file->f_op->write(file, buf, count, pos);
593             else if (file->f_op->write_iter)
594                     ret = new_sync_write(file, buf, count, pos);
595             else
596                     ret = -EINVAL;

(gdb) disass 0xffffffff812e2513
[..]
   0xffffffff812e250e <+190>:   call   0xffffffff82202bc0 <__x86_indirect_thunk_array>
   0xffffffff812e2513 <+195>:   mov    %rax,%r12
   0xffffffff812e2516 <+198>:   test   %r12,%r12
   0xffffffff812e2519 <+201>:   jg     0xffffffff812e257c <vfs_write+300>


And I can add a kprobe the same way:

 # cd /sys/kernel/tracing
 # echo 'p:write vfs_write+0xc3 a=%ax' > kprobe_events
 # trace-cmd start -e write
 # trace-cmd show
[..]
       trace-cmd-884     [006] d.Z..   563.447396: write: (vfs_write+0xc3/0x2b0) a=0x1
  NetworkManager-461     [000] d.Z..   564.791375: write: (vfs_write+0xc3/0x2b0) a=0x8
  NetworkManager-461     [000] d.Z..   564.791408: write: (vfs_write+0xc3/0x2b0) a=0x8


> That said, /proc/kallsyms always gives the accurate, present symbol addresses. 
> But, to my knowledge, gdb isn't able to import /proc/kallsyms as a symbol table.
> To circumvent this, I've written a small userland tool, usling libbfd, that 
> creates an ELF file out of /proc/kallsyms. Passing this ELF file to gdb instead 
> of "vmlinux", and /proc/kcore as core, then allows for a perfect gdb session on 
> the running kernel. Of course this ELF file is only valid until the next reboot, 
> but that's okay as its creation is fast.
> 
> Now, my question: did I miss an alternative ?
> 
> In other words, is there some kind of "kallsyms plug-in" for gdb somewhere ?
> Or, taking the problem from the other side, some kernel module exposing a 
> "/proc/kallsyms.elf" or similar, for direct consumption by gdb ?
> Or another method, that people routinely use for the same purpose ?
> 

Do you need the absolute address? Can't you just use the offset of functions?

-- Steve

Re: Ftrace, KASLR and gdb

[alexandre.ferrieux]

On 10/05/2024 20:12, Steven Rostedt wrote:
> 

Hi Steven. Honoured to be talking to the creator of my preferred toy !

>> Ftrace is a jewel to dig into the kernel, be it for troubleshooting, perf tuning 
>> or just understanding.
>> But when one needs to disassemble the running kernel (eg to move kprobes around 
>> in a function, in order to understand a given code path), KASLR makes it 
>> impossible for gdb to get useful symbol addresses, even with a debug image.
> 
> Really? Can't you just use a function name plus offset? For instance, I
> do this all the time:

Yes, on the ftrace side, obviously name+offset are sufficient. I use them all 
the time, and am delighted with fetchargs.

The problem is rather, on the gdb side, with a production kernel (not the one 
you've just compiled), on a random machine you have a passing access to. First, 
the kernel itself is compressed (vmlinuz), and in many cases getting the vmlinux 
is a pain. So, what ELF binary do you pass to gdb ?

In my case:

   sym2elf /proc/kallsyms > /tmp/kallsyms.elf
   gdb /tmp/kallsyms.elf /proc/kcore
   (gdb) disass vfs_write
   ...

As an added bonus, I see (a snapshot of) globals:

   (gdb) p *(int *)jiffies
   $1 = 92299928

And also I see ftrace's surgical changes if by accident I disassemble a function 
I currently am sniffing ;-)

    echo "p:kprobe_tpacket_rcv tpacket_rcv+26 skb=%di:x64" >> kprobe_events
    (gdb) disass tcpaket_rcv
    Dump of assembler code for function tpacket_rcv:
      0xffffffffb9d79e10 <+0>:     endbr64
      ...
      0xffffffffb9d79e2a <+26>:    jmp    0xffffffffc1389000 <kprobe_optinsn_page>

Of course the latter only serves my curiosity; but it shows I'm looking at the 
"live" code. It matters in other cases of dynamic code, like static branches.

So, is there currently another method to reach the same effect ?

-Alex
____________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

Re: Ftrace, KASLR and gdb

[Steven Rostedt]

On Sun, 12 May 2024 00:44:27 +0200
alexandre.ferrieux@orange.com wrote:

> 
> The problem is rather, on the gdb side, with a production kernel (not the one 
> you've just compiled), on a random machine you have a passing access to. First, 
> the kernel itself is compressed (vmlinuz), and in many cases getting the vmlinux 
> is a pain. So, what ELF binary do you pass to gdb ?
> 
> In my case:
> 
>    sym2elf /proc/kallsyms > /tmp/kallsyms.elf
>    gdb /tmp/kallsyms.elf /proc/kcore
>    (gdb) disass vfs_write

Ah, so you are running gdb on the live kernel.

>    ...
> 
> As an added bonus, I see (a snapshot of) globals:
> 
>    (gdb) p *(int *)jiffies
>    $1 = 92299928
> 
> And also I see ftrace's surgical changes if by accident I disassemble a function 
> I currently am sniffing ;-)
> 
>     echo "p:kprobe_tpacket_rcv tpacket_rcv+26 skb=%di:x64" >> kprobe_events
>     (gdb) disass tcpaket_rcv
>     Dump of assembler code for function tpacket_rcv:
>       0xffffffffb9d79e10 <+0>:     endbr64
>       ...
>       0xffffffffb9d79e2a <+26>:    jmp    0xffffffffc1389000 <kprobe_optinsn_page>
> 
> Of course the latter only serves my curiosity; but it shows I'm looking at the 
> "live" code. It matters in other cases of dynamic code, like static branches.

Yep.

> 
> So, is there currently another method to reach the same effect ?

Besides kgdb (usually another machine connected to the live kernel),
this is probably the only way to do what you want.

-- Steve

Re: Ftrace, KASLR and gdb

[alexandre.ferrieux]

On 13/05/2024 18:25, Steven Rostedt wrote:
> 
> Ah, so you are running gdb on the live kernel.

Yep - sorry I didn't make it clear in the first place :)

>> So, is there currently another method to reach the same effect ?
> 
> Besides kgdb (usually another machine connected to the live kernel),
> this is probably the only way to do what you want.
Ok, thank you for the info.

One reason I'm not using kgdb, beside the obvious pain it is to set up a second 
machine with the appropriate link... is the spirit of ftrace itself: no need for 
breakpoints when you have a nearly-zero-overhead, flexible logger.

And learning from the "live beast" is a 1000x speedup wrt reading the source. 
Learning the Linux Kernel, as open as its source may be, largely remains a 
reverse-engineering operation. With Ftrace, Pahole, and Gdb, one can dig 
*anywhere*. My reasserted kudos for making this possible !

-Alex



____________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.