QEMU function trace

QEMU function trace

[wanghw364]

[-- Attachment #1: Type: text/plain, Size: 602 bytes --]




Hi all,


Does qemu-system-riscv64 have any plugin or tools that can support target program function trace feature?


It seems there is no such feature under link:https://gitlab.com/qemu-project/qemu/-/blob/master/docs/devel/tcg-plugins.rst


For example, we can use libexeclog.so plugin to trace target program instruction trace.


In my case, when I boot linux kernel with qemu, it hangs in the halfway, but I don't know the hang position in the code, 


so I want to trace the kernel function calling trace so that I can find out when and where execution diverges.






Thanks. 

















[-- Attachment #2: Type: text/html, Size: 1620 bytes --]

Re: QEMU function trace

[Alex Bennée]

wanghw364 <wanghw364@163.com> writes:

> Hi all,
>
> Does qemu-system-riscv64 have any plugin or tools that can support target program function trace feature?
>
> It seems there is no such feature under
> link:https://gitlab.com/qemu-project/qemu/-/blob/master/docs/devel/tcg-plugins.rst 
>
> For example, we can use libexeclog.so plugin to trace target program instruction trace.
>
> In my case, when I boot linux kernel with qemu, it hangs in the halfway, but I don't know the hang position in
> the code, 
>
> so I want to trace the kernel function calling trace so that I can
> find out when and where execution diverges.

Not currently but it wouldn't be super hard to write such a thing.
However currently we only have debug symbols available for linux-user so
that is all the helper qemu_plugin_insn_symbol() will see.

You need to teach the linux kernel loader to understand and relocate
symbols from an ELF kernel image. Alternatively you could extract then
and feed them directly to the plugin. It would then be fairly trivial to
stick an execution callback at every function entrance.

I suspect KASLR messes things up though.

>
> Thanks. 


-- 
Alex Bennée

Re:Re: QEMU function trace

[wanghw364]

[-- Attachment #1: Type: text/plain, Size: 2885 bytes --]

Thanks. I have several questions as below, please help, thanks.


1.What do you mean by "only have debug symbols available for linux-user so"? What does the linux-user so refer to?
 qemu_plugin_insn_symbol() can only see symbols from linux-user so?


2.The purpose of teaching the linux kernel loader to understand and relocate symbols from an ELF kernel image,
or extract then and feed them directly to the plugin, is to solve the issue that qemu_plugin_insn_symbol() can't see kernel symbol?


3.How to make the kernel loader understand and relocate symbols in QEMU? How to feed the symbol table directly to the plugin?
As I can see, cache plugin has used qemu_plugin_insn_symbol() and there is function name info in the output result,
but it seems there is no symbol table feeding in the command, shown in https://gitlab.com/qemu-project/qemu/-/blob/master/docs/devel/tcg-plugins.rst.
$ qemu-x86_64 -plugin ./contrib/plugins/libcache.so -d plugin -D cache.log ./tests/tcg/x86_64-linux-user/float_convs


So I was wondering how the symbol table was fed into the plugin? What is the usage of para ./tests/tcg/x86_64-linux-user/float_convs?


4.If we make kernel symbol visible to qemu_plugin_insn_symbol(), the only thing we need to do is to make the core model identify which instruction
is the start of one function and record the function trace by looking up symbol table once the function-level start instruction was executed?


Actually I have the kernel symbol table file named 'System.map' under the kernel directory, I was wondering how to feed it to the plugin.






Thanks.





















At 2022-12-13 23:44:29, "Alex Bennée" <alex.bennee@linaro.org> wrote:
>
>wanghw364 <wanghw364@163.com> writes:
>
>> Hi all,
>>
>> Does qemu-system-riscv64 have any plugin or tools that can support target program function trace feature?
>>
>> It seems there is no such feature under
>> link:https://gitlab.com/qemu-project/qemu/-/blob/master/docs/devel/tcg-plugins.rst 
>>
>> For example, we can use libexeclog.so plugin to trace target program instruction trace.
>>
>> In my case, when I boot linux kernel with qemu, it hangs in the halfway, but I don't know the hang position in
>> the code, 
>>
>> so I want to trace the kernel function calling trace so that I can
>> find out when and where execution diverges.
>
>Not currently but it wouldn't be super hard to write such a thing.
>However currently we only have debug symbols available for linux-user so
>that is all the helper qemu_plugin_insn_symbol() will see.
>
>You need to teach the linux kernel loader to understand and relocate
>symbols from an ELF kernel image. Alternatively you could extract then
>and feed them directly to the plugin. It would then be fairly trivial to
>stick an execution callback at every function entrance.
>
>I suspect KASLR messes things up though.
>
>>
>> Thanks. 
>
>
>-- 
>Alex Bennée

[-- Attachment #2: Type: text/html, Size: 6440 bytes --]

Re: QEMU function trace

[Alex Bennée]

wanghw364  <wanghw364@163.com> writes:

> Thanks. I have several questions as below, please help, thanks.
>
> 1.What do you mean by "only have debug symbols available for linux-user so"? What does the linux-user so
> refer to?
>  qemu_plugin_insn_symbol() can only see symbols from linux-user so?

The linux-user ELF loader will read the debug symbols (if they exist)
and populate the syminfos structures that lookup_symbol uses. It's
partially obscured by the ELF loaders heavy use of macros but see:

  static void glue(load_symbols, SZ)(struct elfhdr *ehdr, int fd, int must_swab,
                                     int clear_lsb, symbol_fn_t sym_cb)

in elf_ops.h

> 2.The purpose of teaching the linux kernel loader to understand and relocate symbols from an ELF kernel
> image,
> or extract then and feed them directly to the plugin, is to solve the issue that qemu_plugin_insn_symbol()
> can't see kernel symbol?

Yes. This is slightly complicated by the fact that the kernel loaders don't
expect to load pure ELF files but something that is wrapped up as a
Linux loader. For example:

  ➜  file vmlinux
  vmlinux: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=21166458a10404e6157abf0da4a0921144c72675, with debug_info, not stripped
  🕙10:07:42 alex@zen:linux.git/builds/arm64.initramfs with arm64/aarch64-linux-gnu- on  linux-6.0.y [$!?] 
  ➜  file arch/arm64/boot/Image
  arch/arm64/boot/Image: Linux kernel ARM64 boot executable Image, little-endian, 4K pages

The second file is what is actually passed to -kernel in a typical boot.

The logic in arm_setup_direct_kernel_boot() implies you can load ELFs
directly and boot them but for some reason the Linux kernel doesn't work
if you try this way.

> 3.How to make the kernel loader understand and relocate symbols in QEMU? How to feed the symbol table
> directly to the plugin?
> As I can see, cache plugin has used qemu_plugin_insn_symbol() and there is function name info in the output
> result, 
> but it seems there is no symbol table feeding in the command, shown in
> https://gitlab.com/qemu-project/qemu/-/blob/master/docs/devel/tcg-plugins.rst . 
> $ qemu-x86_64 -plugin ./contrib/plugins/libcache.so -d plugin -D cache.log .
> /tests/tcg/x86_64-linux-user/float_convs
>
> So I was wondering how the symbol table was fed into the plugin? What is the usage of para .
> /tests/tcg/x86_64-linux-user/float_convs?

It came directly from the debug symbols embedded in the ELF binary.

> 4.If we make kernel symbol visible to qemu_plugin_insn_symbol(), the only thing we need to do is to make the
> core model identify which instruction
> is the start of one function and record the function trace by looking up symbol table once the function-level
> start instruction was executed?
>
> Actually I have the kernel symbol table file named 'System.map' under the kernel directory, I was wondering
> how to feed it to the plugin.

You could certainly write a System.map parser in your plugin and get the
addresses from that instead. It would probably be faster than working
out what to fix in the kernel load path.

>
> Thanks.
>
> At 2022-12-13 23:44:29, "Alex Bennée" <alex.bennee@linaro.org> wrote:
>>
>>wanghw364 <wanghw364@163.com> writes:
>>
>>> Hi all,
>>>
>>> Does qemu-system-riscv64 have any plugin or tools that can support target program function trace feature?
>>>
>>> It seems there is no such feature under
>>> link:https://gitlab.com/qemu-project/qemu/-/blob/master/docs/devel/tcg-plugins.rst 
>>>
>>> For example, we can use libexeclog.so plugin to trace target program instruction trace.
>>>
>>> In my case, when I boot linux kernel with qemu, it hangs in the halfway, but I don't know the hang position in
>>> the code, 
>>>
>>> so I want to trace the kernel function calling trace so that I can
>>> find out when and where execution diverges.
>>
>>Not currently but it wouldn't be super hard to write such a thing.
>>However currently we only have debug symbols available for linux-user so
>>that is all the helper qemu_plugin_insn_symbol() will see.
>>
>>You need to teach the linux kernel loader to understand and relocate
>>symbols from an ELF kernel image. Alternatively you could extract then
>>and feed them directly to the plugin. It would then be fairly trivial to
>>stick an execution callback at every function entrance.
>>
>>I suspect KASLR messes things up though.
>>
>>>
>>> Thanks. 
>>
>>
>>-- 
>>Alex Bennée


-- 
Alex Bennée

Re: QEMU function trace

[Alex Bennée]

Alex Bennée <alex.bennee@linaro.org> writes:

> wanghw364  <wanghw364@163.com> writes:
>
>> Thanks. I have several questions as below, please help, thanks.
>>
>> 1.What do you mean by "only have debug symbols available for linux-user so"? What does the linux-user so
>> refer to?
>>  qemu_plugin_insn_symbol() can only see symbols from linux-user so?
>
> The linux-user ELF loader will read the debug symbols (if they exist)
> and populate the syminfos structures that lookup_symbol uses. It's
> partially obscured by the ELF loaders heavy use of macros but see:
>
>   static void glue(load_symbols, SZ)(struct elfhdr *ehdr, int fd, int must_swab,
>                                      int clear_lsb, symbol_fn_t sym_cb)
>
> in elf_ops.h
>
>> 2.The purpose of teaching the linux kernel loader to understand and relocate symbols from an ELF kernel
>> image,
>> or extract then and feed them directly to the plugin, is to solve
>> the issue that qemu_plugin_insn_symbol()
>> can't see kernel symbol?
>
> Yes. This is slightly complicated by the fact that the kernel loaders don't
> expect to load pure ELF files but something that is wrapped up as a
> Linux loader. For example:
>
>   ➜  file vmlinux
>   vmlinux: ELF 64-bit LSB pie executable, ARM aarch64, version 1
> (SYSV), statically linked,
> BuildID[sha1]=21166458a10404e6157abf0da4a0921144c72675, with
> debug_info, not stripped
>   🕙10:07:42 alex@zen:linux.git/builds/arm64.initramfs with
> arm64/aarch64-linux-gnu- on  linux-6.0.y [$!?]
>   ➜  file arch/arm64/boot/Image
>   arch/arm64/boot/Image: Linux kernel ARM64 boot executable Image, little-endian, 4K pages
>
> The second file is what is actually passed to -kernel in a typical boot.
>
> The logic in arm_setup_direct_kernel_boot() implies you can load ELFs
> directly and boot them but for some reason the Linux kernel doesn't work
> if you try this way.

Replying to myself - this is because the vmlinux image is based of
kernel virtual address. So the import thing the loader does is create
the initial vaddr mappings and relocate the kernel to that location
before running it. See the function primary_entry in head.S in the
kernel.

So perhaps for system emulation it would be useful to have a -symbols
option to load symbols from another file.

-- 
Alex Bennée

Re: QEMU function trace

[Claudio Fontana]

On 12/14/22 12:00, Alex Bennée wrote:
> 
> Alex Bennée <alex.bennee@linaro.org> writes:
> 
>> wanghw364  <wanghw364@163.com> writes:
>>
>>> Thanks. I have several questions as below, please help, thanks.
>>>
>>> 1.What do you mean by "only have debug symbols available for linux-user so"? What does the linux-user so
>>> refer to?
>>>  qemu_plugin_insn_symbol() can only see symbols from linux-user so?
>>
>> The linux-user ELF loader will read the debug symbols (if they exist)
>> and populate the syminfos structures that lookup_symbol uses. It's
>> partially obscured by the ELF loaders heavy use of macros but see:
>>
>>   static void glue(load_symbols, SZ)(struct elfhdr *ehdr, int fd, int must_swab,
>>                                      int clear_lsb, symbol_fn_t sym_cb)
>>
>> in elf_ops.h
>>
>>> 2.The purpose of teaching the linux kernel loader to understand and relocate symbols from an ELF kernel
>>> image,
>>> or extract then and feed them directly to the plugin, is to solve
>>> the issue that qemu_plugin_insn_symbol()
>>> can't see kernel symbol?
>>
>> Yes. This is slightly complicated by the fact that the kernel loaders don't
>> expect to load pure ELF files but something that is wrapped up as a
>> Linux loader. For example:
>>
>>   ➜  file vmlinux
>>   vmlinux: ELF 64-bit LSB pie executable, ARM aarch64, version 1
>> (SYSV), statically linked,
>> BuildID[sha1]=21166458a10404e6157abf0da4a0921144c72675, with
>> debug_info, not stripped
>>   🕙10:07:42 alex@zen:linux.git/builds/arm64.initramfs with
>> arm64/aarch64-linux-gnu- on  linux-6.0.y [$!?]
>>   ➜  file arch/arm64/boot/Image
>>   arch/arm64/boot/Image: Linux kernel ARM64 boot executable Image, little-endian, 4K pages
>>
>> The second file is what is actually passed to -kernel in a typical boot.
>>
>> The logic in arm_setup_direct_kernel_boot() implies you can load ELFs
>> directly and boot them but for some reason the Linux kernel doesn't work
>> if you try this way.
> 
> Replying to myself - this is because the vmlinux image is based of
> kernel virtual address. So the import thing the loader does is create
> the initial vaddr mappings and relocate the kernel to that location
> before running it. See the function primary_entry in head.S in the
> kernel.
> 
> So perhaps for system emulation it would be useful to have a -symbols
> option to load symbols from another file.
> 

Hi Alex,

it doesn't need to be a tcg plugin-only feature right, it's possible to use qemu to debug the guest also when using KVM..

Ciao,

Claudio

Re: QEMU function trace

[Alex Bennée]

Claudio Fontana <cfontana@suse.de> writes:

> On 12/14/22 12:00, Alex Bennée wrote:
>> 
>> Alex Bennée <alex.bennee@linaro.org> writes:
>> 
>>> wanghw364  <wanghw364@163.com> writes:
>>>
>>>> Thanks. I have several questions as below, please help, thanks.
>>>>
>>>> 1.What do you mean by "only have debug symbols available for
>>>> linux-user so"? What does the linux-user so
>>>> refer to?
>>>>  qemu_plugin_insn_symbol() can only see symbols from linux-user so?
>>>
>>> The linux-user ELF loader will read the debug symbols (if they exist)
>>> and populate the syminfos structures that lookup_symbol uses. It's
>>> partially obscured by the ELF loaders heavy use of macros but see:
>>>
>>>   static void glue(load_symbols, SZ)(struct elfhdr *ehdr, int fd, int must_swab,
>>>                                      int clear_lsb, symbol_fn_t sym_cb)
>>>
>>> in elf_ops.h
>>>
>>>> 2.The purpose of teaching the linux kernel loader to understand and relocate symbols from an ELF kernel
>>>> image,
>>>> or extract then and feed them directly to the plugin, is to solve
>>>> the issue that qemu_plugin_insn_symbol()
>>>> can't see kernel symbol?
>>>
>>> Yes. This is slightly complicated by the fact that the kernel loaders don't
>>> expect to load pure ELF files but something that is wrapped up as a
>>> Linux loader. For example:
>>>
>>>   ➜  file vmlinux
>>>   vmlinux: ELF 64-bit LSB pie executable, ARM aarch64, version 1
>>> (SYSV), statically linked,
>>> BuildID[sha1]=21166458a10404e6157abf0da4a0921144c72675, with
>>> debug_info, not stripped
>>>   🕙10:07:42 alex@zen:linux.git/builds/arm64.initramfs with
>>> arm64/aarch64-linux-gnu- on  linux-6.0.y [$!?]
>>>   ➜  file arch/arm64/boot/Image
>>>   arch/arm64/boot/Image: Linux kernel ARM64 boot executable Image, little-endian, 4K pages
>>>
>>> The second file is what is actually passed to -kernel in a typical boot.
>>>
>>> The logic in arm_setup_direct_kernel_boot() implies you can load ELFs
>>> directly and boot them but for some reason the Linux kernel doesn't work
>>> if you try this way.
>> 
>> Replying to myself - this is because the vmlinux image is based of
>> kernel virtual address. So the import thing the loader does is create
>> the initial vaddr mappings and relocate the kernel to that location
>> before running it. See the function primary_entry in head.S in the
>> kernel.
>> 
>> So perhaps for system emulation it would be useful to have a -symbols
>> option to load symbols from another file.
>> 
>
> Hi Alex,
>
> it doesn't need to be a tcg plugin-only feature right, it's possible
> to use qemu to debug the guest also when using KVM..

No it doesn't have to be although the only real users is the -d in_asm
trace which I suspect doesn't get used as much for system emulation
given the size of the traces.

For normal debugging over the gdbstub its usually left up to the gdb
process itself to handle the resolution of symbols, e.g.:

  gdb vmlinux -ex "target remote localhost:1234"

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re:Re: QEMU function trace

[wanghw364]

[-- Attachment #1: Type: text/plain, Size: 4113 bytes --]

Thanks very much.


According to your reply, I reviewd qemu loader and plugins source code again, I think I've got what you mentioned.


In qemu linux-user(user mode) case, the guest images are all ELF format, and inside the ELF loader qemu will call load_symbols function, so in this case, the symbol table is loaded while loading ELF automatically, we don't need extra effort to load symbol table.


But for kernel booting case, the guest image format we use in -kernel option is Image but not ELF, and there is no load_symbols behavior during the loading process, so plugins can't see the kernel symbol.


What we probably need to do is adding symbol loading behavior in kernel loader and populating syminfos to plugins, or writing a System.map parser in self-defined plugin as you said, maybe the later is a faster way.


BTW, sorry for the auto-reply, it has been turned-off, thanks for the heads up.








Thanks.











On 2022-12-14 23:03:38，"Alex Bennée" <alex.bennee@linaro.org> wrote:
>
>Claudio Fontana <cfontana@suse.de> writes:
>
>> On 12/14/22 12:00, Alex Bennée wrote:
>>> 
>>> Alex Bennée <alex.bennee@linaro.org> writes:
>>> 
>>>> wanghw364  <wanghw364@163.com> writes:
>>>>
>>>>> Thanks. I have several questions as below, please help, thanks.
>>>>>
>>>>> 1.What do you mean by "only have debug symbols available for
>>>>> linux-user so"? What does the linux-user so
>>>>> refer to?
>>>>>  qemu_plugin_insn_symbol() can only see symbols from linux-user so?
>>>>
>>>> The linux-user ELF loader will read the debug symbols (if they exist)
>>>> and populate the syminfos structures that lookup_symbol uses. It's
>>>> partially obscured by the ELF loaders heavy use of macros but see:
>>>>
>>>>   static void glue(load_symbols, SZ)(struct elfhdr *ehdr, int fd, int must_swab,
>>>>                                      int clear_lsb, symbol_fn_t sym_cb)
>>>>
>>>> in elf_ops.h
>>>>
>>>>> 2.The purpose of teaching the linux kernel loader to understand and relocate symbols from an ELF kernel
>>>>> image,
>>>>> or extract then and feed them directly to the plugin, is to solve
>>>>> the issue that qemu_plugin_insn_symbol()
>>>>> can't see kernel symbol?
>>>>
>>>> Yes. This is slightly complicated by the fact that the kernel loaders don't
>>>> expect to load pure ELF files but something that is wrapped up as a
>>>> Linux loader. For example:
>>>>
>>>>   ➜  file vmlinux
>>>>   vmlinux: ELF 64-bit LSB pie executable, ARM aarch64, version 1
>>>> (SYSV), statically linked,
>>>> BuildID[sha1]=21166458a10404e6157abf0da4a0921144c72675, with
>>>> debug_info, not stripped
>>>>   10:07:42 alex@zen:linux.git/builds/arm64.initramfs with
>>>> arm64/aarch64-linux-gnu- on  linux-6.0.y [$!?]
>>>>   ➜  file arch/arm64/boot/Image
>>>>   arch/arm64/boot/Image: Linux kernel ARM64 boot executable Image, little-endian, 4K pages
>>>>
>>>> The second file is what is actually passed to -kernel in a typical boot.
>>>>
>>>> The logic in arm_setup_direct_kernel_boot() implies you can load ELFs
>>>> directly and boot them but for some reason the Linux kernel doesn't work
>>>> if you try this way.
>>> 
>>> Replying to myself - this is because the vmlinux image is based of
>>> kernel virtual address. So the import thing the loader does is create
>>> the initial vaddr mappings and relocate the kernel to that location
>>> before running it. See the function primary_entry in head.S in the
>>> kernel.
>>> 
>>> So perhaps for system emulation it would be useful to have a -symbols
>>> option to load symbols from another file.
>>> 
>>
>> Hi Alex,
>>
>> it doesn't need to be a tcg plugin-only feature right, it's possible
>> to use qemu to debug the guest also when using KVM..
>
>No it doesn't have to be although the only real users is the -d in_asm
>trace which I suspect doesn't get used as much for system emulation
>given the size of the traces.
>
>For normal debugging over the gdbstub its usually left up to the gdb
>process itself to handle the resolution of symbols, e.g.:
>
>  gdb vmlinux -ex "target remote localhost:1234"
>
>-- 
>Alex Bennée
>Virtualisation Tech Lead @ Linaro

[-- Attachment #2: Type: text/html, Size: 6298 bytes --]