[Qemu-devel] memory trace with qemu

[Qemu-devel] memory trace with qemu

[Eduardo Cruz]

[-- Attachment #1: Type: text/plain, Size: 672 bytes --]

Hello, I need qemu to keep track of all the memory access made by the guest,
including read, write and the instruction fetches.
I was using Simics to acomplish this task, but it is very slow.
I need the virtual address, the size of the operation, the cycle number when
it happened and if it is a read, write or instruction fetch.
Reading the mail list archives, I saw that the file softmmu_header.h is
always called when a read/write happens.
Is that so? Any ideas of how I can record these information with qemu?

Thanks in advance

-- 
Eduardo Henrique Molina da Cruz
MSc student
Parallel and Distributed Processing Group
Federal University of Rio Grande do Sul (UFRGS)

[-- Attachment #2: Type: text/html, Size: 832 bytes --]

Re: [Qemu-devel] memory trace with qemu

[Mulyadi Santosa]

Hi...

On Fri, Jul 23, 2010 at 05:30, Eduardo Cruz <eduardohmdacruz@gmail.com> wrote:
> Hello, I need qemu to keep track of all the memory access made by the guest,
> including read, write and the instruction fetches.

AFAIK there are lots of experiments on this and has produces working
patches...at least from the posting of the creator. There is even a
patch floating to start creating trace framework a while ago.

perhaps you could dig a little deeper in qemu archieve....?

-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

Re: [Qemu-devel] memory trace with qemu

[Stefan Hajnoczi]

On Fri, Jul 23, 2010 at 4:12 AM, Mulyadi Santosa
<mulyadi.santosa@gmail.com> wrote:
> Hi...
>
> On Fri, Jul 23, 2010 at 05:30, Eduardo Cruz <eduardohmdacruz@gmail.com> wrote:
>> Hello, I need qemu to keep track of all the memory access made by the guest,
>> including read, write and the instruction fetches.
>
> AFAIK there are lots of experiments on this and has produces working
> patches...at least from the posting of the creator. There is even a
> patch floating to start creating trace framework a while ago.

Thanks Mulyadi, I think you are referring to the tracing work that
Prerna Saxena and I are doing.  Here is the documentation:

http://repo.or.cz/w/qemu/stefanha.git/blob/tracing:/docs/tracing.txt

The patches apply to qemu.git.  You can define trace events in the
trace-events file and then call them from places in the code.  There
is a script to pretty-print the binary trace file that QEMU produces.

Eduardo, if you think this might be what you're looking for, please
give it a try.  I am on #qemu and #kvm IRC if you need any help.  Any
feedback will be valuable to us as we prepare these patches for
submission to qemu.git.

I believe the tracing framework answers the "Any ideas of how I can
record these information with qemu?" part of your question :).  I
don't have experience in the TCG, so I can't give advice on how to
best get at the memory accesses, but I hope this helps you one step
further.

Stefan

> perhaps you could dig a little deeper in qemu archieve....?
>
> --
> regards,
>
> Mulyadi Santosa
> Freelance Linux trainer and consultant
>
> blog: the-hydra.blogspot.com
> training: mulyaditraining.blogspot.com
>
>

Re: [Qemu-devel] memory trace with qemu

[Mulyadi Santosa]

On Fri, Jul 23, 2010 at 13:15, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> Thanks Mulyadi, I think you are referring to the tracing work that
> Prerna Saxena and I are doing.  Here is the documentation:
>
> http://repo.or.cz/w/qemu/stefanha.git/blob/tracing:/docs/tracing.txt

You're welcome Stefan :) I just hate to see people reinvent the wheel,
even for "holy and sacred" purpose :)

This makes me believe that Qemu is indeed an (almost) ideal platform
to do system tracing, given that it is open source and support many
many cpu architecture. Possibly in the future (or now?), it would be a
standard for binary analysis platform.

my own question: any benchmark to show us the overhead of each of
these tracing frameworks/patches?

-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

Re: [Qemu-devel] memory trace with qemu

[Stefan Hajnoczi]

On Fri, Jul 23, 2010 at 8:25 AM, Mulyadi Santosa
<mulyadi.santosa@gmail.com> wrote:
> On Fri, Jul 23, 2010 at 13:15, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> Thanks Mulyadi, I think you are referring to the tracing work that
>> Prerna Saxena and I are doing.  Here is the documentation:
>>
>> http://repo.or.cz/w/qemu/stefanha.git/blob/tracing:/docs/tracing.txt
>
> You're welcome Stefan :) I just hate to see people reinvent the wheel,
> even for "holy and sacred" purpose :)
>
> This makes me believe that Qemu is indeed an (almost) ideal platform
> to do system tracing, given that it is open source and support many
> many cpu architecture. Possibly in the future (or now?), it would be a
> standard for binary analysis platform.
>
> my own question: any benchmark to show us the overhead of each of
> these tracing frameworks/patches?

It depends on the trace backend you choose.  Currently there is the
"simple" backend, which is portable and built into the source tree to
offer core functionality, and the "ust" backend, which uses the LTTng
Userspace Tracer for fancier but fussier tracing on Linux.

The performance of these backends is different.  If performance is a
top concern, then the "ust" backend is probably the way to go.  See
http://lttng.org/ust for more info.

However, I have successfully used the simple backend for block layer
latency tracing: http://www.linux-kvm.org/page/Virtio/Block/Latency.

Stefan

> --
> regards,
>
> Mulyadi Santosa
> Freelance Linux trainer and consultant
>
> blog: the-hydra.blogspot.com
> training: mulyaditraining.blogspot.com
>

Re: [Qemu-devel] memory trace with qemu

[Mulyadi Santosa]

On Fri, Jul 23, 2010 at 15:39, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> The performance of these backends is different.  If performance is a
> top concern, then the "ust" backend is probably the way to go.  See
> http://lttng.org/ust for more info.


Oh wow, so let me put it another way: it's using LTTng? neat!

-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

Re: [Qemu-devel] memory trace with qemu

[Jun Koi]

On Fri, Jul 23, 2010 at 3:15 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Fri, Jul 23, 2010 at 4:12 AM, Mulyadi Santosa
> <mulyadi.santosa@gmail.com> wrote:
>> Hi...
>>
>> On Fri, Jul 23, 2010 at 05:30, Eduardo Cruz <eduardohmdacruz@gmail.com> wrote:
>>> Hello, I need qemu to keep track of all the memory access made by the guest,
>>> including read, write and the instruction fetches.
>>
>> AFAIK there are lots of experiments on this and has produces working
>> patches...at least from the posting of the creator. There is even a
>> patch floating to start creating trace framework a while ago.
>
> Thanks Mulyadi, I think you are referring to the tracing work that
> Prerna Saxena and I are doing.  Here is the documentation:
>
> http://repo.or.cz/w/qemu/stefanha.git/blob/tracing:/docs/tracing.txt
>
> The patches apply to qemu.git.  You can define trace events in the
> trace-events file and then call them from places in the code.  There
> is a script to pretty-print the binary trace file that QEMU produces.
>

what trace events do you support now?

thanks,
J

Re: [Qemu-devel] memory trace with qemu

[Stefan Hajnoczi]

On Fri, Jul 23, 2010 at 8:34 AM, Jun Koi <junkoi2004@gmail.com> wrote:
> On Fri, Jul 23, 2010 at 3:15 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Fri, Jul 23, 2010 at 4:12 AM, Mulyadi Santosa
>> <mulyadi.santosa@gmail.com> wrote:
>>> Hi...
>>>
>>> On Fri, Jul 23, 2010 at 05:30, Eduardo Cruz <eduardohmdacruz@gmail.com> wrote:
>>>> Hello, I need qemu to keep track of all the memory access made by the guest,
>>>> including read, write and the instruction fetches.
>>>
>>> AFAIK there are lots of experiments on this and has produces working
>>> patches...at least from the posting of the creator. There is even a
>>> patch floating to start creating trace framework a while ago.
>>
>> Thanks Mulyadi, I think you are referring to the tracing work that
>> Prerna Saxena and I are doing.  Here is the documentation:
>>
>> http://repo.or.cz/w/qemu/stefanha.git/blob/tracing:/docs/tracing.txt
>>
>> The patches apply to qemu.git.  You can define trace events in the
>> trace-events file and then call them from places in the code.  There
>> is a script to pretty-print the binary trace file that QEMU produces.
>>
>
> what trace events do you support now?

The existing set of trace events is here:

http://repo.or.cz/w/qemu/stefanha.git/blob/tracing:/trace-events

It is a limited set of pre-defined trace events - most come from my
block latency tracing work.  But adding trace events is easy.

There is a task on the roadmap to get broader tracing coverage before
submitting the patches to qemu.git.  If you add trace events that you
feel would be useful to others, please don't hesitate to send patches
so they can be included.

Stefan

> thanks,
> J
>

Re: [Qemu-devel] memory trace with qemu

[Yufei Chen]

On Fri, Jul 23, 2010 at 2:15 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Fri, Jul 23, 2010 at 4:12 AM, Mulyadi Santosa
> <mulyadi.santosa@gmail.com> wrote:
>> Hi...
>>
>> On Fri, Jul 23, 2010 at 05:30, Eduardo Cruz <eduardohmdacruz@gmail.com> wrote:
>>> Hello, I need qemu to keep track of all the memory access made by the guest,
>>> including read, write and the instruction fetches.

I don't think Qemu can provide cycle number information. But other
information are all available.

>>
>> AFAIK there are lots of experiments on this and has produces working
>> patches...at least from the posting of the creator. There is even a
>> patch floating to start creating trace framework a while ago.
>
> Thanks Mulyadi, I think you are referring to the tracing work that
> Prerna Saxena and I are doing.  Here is the documentation:
>
> http://repo.or.cz/w/qemu/stefanha.git/blob/tracing:/docs/tracing.txt
>
> The patches apply to qemu.git.  You can define trace events in the
> trace-events file and then call them from places in the code.  There
> is a script to pretty-print the binary trace file that QEMU produces.
>
> Eduardo, if you think this might be what you're looking for, please
> give it a try.  I am on #qemu and #kvm IRC if you need any help.  Any
> feedback will be valuable to us as we prepare these patches for
> submission to qemu.git.
>
> I believe the tracing framework answers the "Any ideas of how I can
> record these information with qemu?" part of your question :).  I
> don't have experience in the TCG, so I can't give advice on how to
> best get at the memory accesses, but I hope this helps you one step
> further.
>

Can this framework trace memory access event? I guess this would be
more difficult to do in KVM than in TCG.

> Stefan
>
>> perhaps you could dig a little deeper in qemu archieve....?
>>
>> --
>> regards,
>>
>> Mulyadi Santosa
>> Freelance Linux trainer and consultant
>>
>> blog: the-hydra.blogspot.com
>> training: mulyaditraining.blogspot.com
>>
>>
>
>



-- 
Best regards,
Chen Yufei

Re: [Qemu-devel] memory trace with qemu

[Stefan Hajnoczi]

On Sat, Jul 24, 2010 at 3:17 AM, Yufei Chen <cyfdecyf@gmail.com> wrote:
> On Fri, Jul 23, 2010 at 2:15 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Fri, Jul 23, 2010 at 4:12 AM, Mulyadi Santosa
>> <mulyadi.santosa@gmail.com> wrote:
>>> Hi...
>>>
>>> On Fri, Jul 23, 2010 at 05:30, Eduardo Cruz <eduardohmdacruz@gmail.com> wrote:
>>>> Hello, I need qemu to keep track of all the memory access made by the guest,
>>>> including read, write and the instruction fetches.
>
> I don't think Qemu can provide cycle number information. But other
> information are all available.
>
>>>
>>> AFAIK there are lots of experiments on this and has produces working
>>> patches...at least from the posting of the creator. There is even a
>>> patch floating to start creating trace framework a while ago.
>>
>> Thanks Mulyadi, I think you are referring to the tracing work that
>> Prerna Saxena and I are doing.  Here is the documentation:
>>
>> http://repo.or.cz/w/qemu/stefanha.git/blob/tracing:/docs/tracing.txt
>>
>> The patches apply to qemu.git.  You can define trace events in the
>> trace-events file and then call them from places in the code.  There
>> is a script to pretty-print the binary trace file that QEMU produces.
>>
>> Eduardo, if you think this might be what you're looking for, please
>> give it a try.  I am on #qemu and #kvm IRC if you need any help.  Any
>> feedback will be valuable to us as we prepare these patches for
>> submission to qemu.git.
>>
>> I believe the tracing framework answers the "Any ideas of how I can
>> record these information with qemu?" part of your question :).  I
>> don't have experience in the TCG, so I can't give advice on how to
>> best get at the memory accesses, but I hope this helps you one step
>> further.
>>
>
> Can this framework trace memory access event? I guess this would be
> more difficult to do in KVM than in TCG.

No, it can't trace memory accesses.  It provides you with the ability
to put trace events into QEMU, but there is current no pre-defined
trace event for memory access.

If you find the right spot to in TCG it should be easy to add a trace
event there.  I agree, with KVM is would be more difficult and perhaps
defeat the point of KVM :).

Stefan

>> Stefan
>>
>>> perhaps you could dig a little deeper in qemu archieve....?
>>>
>>> --
>>> regards,
>>>
>>> Mulyadi Santosa
>>> Freelance Linux trainer and consultant
>>>
>>> blog: the-hydra.blogspot.com
>>> training: mulyaditraining.blogspot.com
>>>
>>>
>>
>>
>
>
>
> --
> Best regards,
> Chen Yufei
>

Re: [Qemu-devel] memory trace with qemu

[Eduardo Cruz]

[-- Attachment #1: Type: text/plain, Size: 3524 bytes --]

Thanks for your awnsers. Stean, after I find the right place to capture the
reads and writes I'll definitely try your trace tool.

Until now, this is what i found:

I am using the x86-64 target, and I know that, for instance, lots of reads
pass here:

target-i386/translate.c   gen_op_ld_T1_A0()

This function calls gen_op_ld_v, that calls a tcg function according to the
operand size.
Lets assume tcg_gen_qemu_ld8u() is called, which is defined at tcg/tcg-op.h

Here I find myself lost again, because this function (in i386)
calls tcg_gen_op3i_i32, which APPARENTLY does nothing...

Where the code that access the main memory is being generated?

2010/7/24 Stefan Hajnoczi <stefanha@gmail.com>

> On Sat, Jul 24, 2010 at 3:17 AM, Yufei Chen <cyfdecyf@gmail.com> wrote:
> > On Fri, Jul 23, 2010 at 2:15 PM, Stefan Hajnoczi <stefanha@gmail.com>
> wrote:
> >> On Fri, Jul 23, 2010 at 4:12 AM, Mulyadi Santosa
> >> <mulyadi.santosa@gmail.com> wrote:
> >>> Hi...
> >>>
> >>> On Fri, Jul 23, 2010 at 05:30, Eduardo Cruz <eduardohmdacruz@gmail.com>
> wrote:
> >>>> Hello, I need qemu to keep track of all the memory access made by the
> guest,
> >>>> including read, write and the instruction fetches.
> >
> > I don't think Qemu can provide cycle number information. But other
> > information are all available.
> >
> >>>
> >>> AFAIK there are lots of experiments on this and has produces working
> >>> patches...at least from the posting of the creator. There is even a
> >>> patch floating to start creating trace framework a while ago.
> >>
> >> Thanks Mulyadi, I think you are referring to the tracing work that
> >> Prerna Saxena and I are doing.  Here is the documentation:
> >>
> >> http://repo.or.cz/w/qemu/stefanha.git/blob/tracing:/docs/tracing.txt
> >>
> >> The patches apply to qemu.git.  You can define trace events in the
> >> trace-events file and then call them from places in the code.  There
> >> is a script to pretty-print the binary trace file that QEMU produces.
> >>
> >> Eduardo, if you think this might be what you're looking for, please
> >> give it a try.  I am on #qemu and #kvm IRC if you need any help.  Any
> >> feedback will be valuable to us as we prepare these patches for
> >> submission to qemu.git.
> >>
> >> I believe the tracing framework answers the "Any ideas of how I can
> >> record these information with qemu?" part of your question :).  I
> >> don't have experience in the TCG, so I can't give advice on how to
> >> best get at the memory accesses, but I hope this helps you one step
> >> further.
> >>
> >
> > Can this framework trace memory access event? I guess this would be
> > more difficult to do in KVM than in TCG.
>
> No, it can't trace memory accesses.  It provides you with the ability
> to put trace events into QEMU, but there is current no pre-defined
> trace event for memory access.
>
> If you find the right spot to in TCG it should be easy to add a trace
> event there.  I agree, with KVM is would be more difficult and perhaps
> defeat the point of KVM :).
>
> Stefan
>
> >> Stefan
> >>
> >>> perhaps you could dig a little deeper in qemu archieve....?
> >>>
> >>> --
> >>> regards,
> >>>
> >>> Mulyadi Santosa
> >>> Freelance Linux trainer and consultant
> >>>
> >>> blog: the-hydra.blogspot.com
> >>> training: mulyaditraining.blogspot.com
> >>>
> >>>
> >>
> >>
> >
> >
> >
> > --
> > Best regards,
> > Chen Yufei
> >
>



-- 
Eduardo Henrique Molina da Cruz
MSc student
Parallel and Distributed Processing Group
Federal University of Rio Grande do Sul (UFRGS)

[-- Attachment #2: Type: text/html, Size: 5218 bytes --]

Re: [Qemu-devel] memory trace with qemu

[Yufei Chen]

On Mon, Jul 26, 2010 at 6:21 AM, Eduardo Cruz <eduardohmdacruz@gmail.com> wrote:
> Thanks for your awnsers. Stean, after I find the right place to capture the
> reads and writes I'll definitely try your trace tool.
> Until now, this is what i found:
>
> I am using the x86-64 target, and I know that, for instance, lots of reads
> pass here:
> target-i386/translate.c   gen_op_ld_T1_A0()
> This function calls gen_op_ld_v, that calls a tcg function according to the
> operand size.
> Lets assume tcg_gen_qemu_ld8u() is called, which is defined at tcg/tcg-op.h
> Here I find myself lost again, because this function (in i386)
> calls tcg_gen_op3i_i32, which APPARENTLY does nothing...
> Where the code that access the main memory is being generated?

Please take a look at tcg_out_qemu_ld and tcg_out_qemu_st, also take a
look at tcg_out_op.

> 2010/7/24 Stefan Hajnoczi <stefanha@gmail.com>
>>
>> On Sat, Jul 24, 2010 at 3:17 AM, Yufei Chen <cyfdecyf@gmail.com> wrote:
>> > On Fri, Jul 23, 2010 at 2:15 PM, Stefan Hajnoczi <stefanha@gmail.com>
>> > wrote:
>> >> On Fri, Jul 23, 2010 at 4:12 AM, Mulyadi Santosa
>> >> <mulyadi.santosa@gmail.com> wrote:
>> >>> Hi...
>> >>>
>> >>> On Fri, Jul 23, 2010 at 05:30, Eduardo Cruz
>> >>> <eduardohmdacruz@gmail.com> wrote:
>> >>>> Hello, I need qemu to keep track of all the memory access made by the
>> >>>> guest,
>> >>>> including read, write and the instruction fetches.
>> >
>> > I don't think Qemu can provide cycle number information. But other
>> > information are all available.
>> >
>> >>>
>> >>> AFAIK there are lots of experiments on this and has produces working
>> >>> patches...at least from the posting of the creator. There is even a
>> >>> patch floating to start creating trace framework a while ago.
>> >>
>> >> Thanks Mulyadi, I think you are referring to the tracing work that
>> >> Prerna Saxena and I are doing.  Here is the documentation:
>> >>
>> >> http://repo.or.cz/w/qemu/stefanha.git/blob/tracing:/docs/tracing.txt
>> >>
>> >> The patches apply to qemu.git.  You can define trace events in the
>> >> trace-events file and then call them from places in the code.  There
>> >> is a script to pretty-print the binary trace file that QEMU produces.
>> >>
>> >> Eduardo, if you think this might be what you're looking for, please
>> >> give it a try.  I am on #qemu and #kvm IRC if you need any help.  Any
>> >> feedback will be valuable to us as we prepare these patches for
>> >> submission to qemu.git.
>> >>
>> >> I believe the tracing framework answers the "Any ideas of how I can
>> >> record these information with qemu?" part of your question :).  I
>> >> don't have experience in the TCG, so I can't give advice on how to
>> >> best get at the memory accesses, but I hope this helps you one step
>> >> further.
>> >>
>> >
>> > Can this framework trace memory access event? I guess this would be
>> > more difficult to do in KVM than in TCG.
>>
>> No, it can't trace memory accesses.  It provides you with the ability
>> to put trace events into QEMU, but there is current no pre-defined
>> trace event for memory access.
>>
>> If you find the right spot to in TCG it should be easy to add a trace
>> event there.  I agree, with KVM is would be more difficult and perhaps
>> defeat the point of KVM :).
>>
>> Stefan
>>
>> >> Stefan
>> >>
>> >>> perhaps you could dig a little deeper in qemu archieve....?
>> >>>
>> >>> --
>> >>> regards,
>> >>>
>> >>> Mulyadi Santosa
>> >>> Freelance Linux trainer and consultant
>> >>>
>> >>> blog: the-hydra.blogspot.com
>> >>> training: mulyaditraining.blogspot.com
>> >>>
>> >>>
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Best regards,
>> > Chen Yufei
>> >
>
>
>
> --
> Eduardo Henrique Molina da Cruz
> MSc student
> Parallel and Distributed Processing Group
> Federal University of Rio Grande do Sul (UFRGS)
>
>



-- 
Best regards,
Chen Yufei

Re: [Qemu-devel] memory trace with qemu

[Lluís]

Eduardo Cruz writes:

> Thanks for your awnsers. Stean, after I find the right place to capture the
> reads and writes I'll definitely try your trace tool.

> Until now, this is what i found:

> I am using the x86-64 target, and I know that, for instance, lots of reads
> pass here:

> target-i386/translate.c   gen_op_ld_T1_A0()

Ok, I've seen at least 3 people working on this lately.

Some time ago I wrote a message proposing two sets of modifications for qemu, in
order to allow the analysis of guest code (like feeding traces to an
architecture simulator).

What I proposed is based on two different functionalities:

1) backdoor: a mechanism for the guest to communicate with qemu, such that
   tracing can be started, stopped, etc.

   My current approach is to decode an instruction that is deemed invalid by the
   target ISA according to the manual.

   This is only implemented for x86 right now, but it is trivial to implement on
   other architectures as long as there are unused opcodes.

2) instrumentation: a set of generic macros that signal events that might be of
   interest.

   The idea is to have the very same macros on all targets, such that the user
   can just define what to do when these are called (like using the qemu tracing
   mechanism), without bothering about the target architecture.

   These include memory accesses, instruction fetch information (address, size,
   used registers, defined registers and a generic opcode type), etc.
   I started implementing it for x86, but the ISA is a hellish nightmare (no
   news), and got sidetracked by other pending work.

   The main idea is that instrumentation can be dynamically and selectively
   activated/deactivated (thus the backdoor mechanism), and it is generated as
   extra TCG code when active.

My intent was to spin up some discussion on which macros to provide, as well as
the most performant mechanism for for the dynamic (de)activation of these.

So, the question is, again, if the qemu community is willing to provide this, or
should we maintain it as a separate set of patches.

As I told Jun Koi, I'll try to set up a git repository at work for others to
have access to the modifications and discuss the alternatives.


Oh! BTW, my current prototype uses code like this on target-x86/translate.c:

    #define tcg_gen_qemu_ld8u(arg, addr, mem_index)          \
        do {                                                 \
            INSTR_MEM(addr, 8);                              \
            (tcg_gen_qemu_ld8u)(arg, addr, mem_index);       \
        } while (0)

Of course, this doesn't take into account DMA memory accesses and accesses
produced by helper code.


Lluís

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: [Qemu-devel] memory trace with qemu

[Eliot Moss]

On 7/26/2010 6:20 AM, Lluís wrote:
> Eduardo Cruz writes:
>
>> Thanks for your awnsers. Stean, after I find the right place to capture the
>> reads and writes I'll definitely try your trace tool.
>
>> Until now, this is what i found:
>
>> I am using the x86-64 target, and I know that, for instance, lots of reads
>> pass here:
>
>> target-i386/translate.c   gen_op_ld_T1_A0()
>
> Ok, I've seen at least 3 people working on this lately.
>
> Some time ago I wrote a message proposing two sets of modifications for qemu, in
> order to allow the analysis of guest code (like feeding traces to an
> architecture simulator).
>
> What I proposed is based on two different functionalities:
>
> 1) backdoor: a mechanism for the guest to communicate with qemu, such that
>     tracing can be started, stopped, etc.
>
>     My current approach is to decode an instruction that is deemed invalid by the
>     target ISA according to the manual.
>
>     This is only implemented for x86 right now, but it is trivial to implement on
>     other architectures as long as there are unused opcodes.
>
> 2) instrumentation: a set of generic macros that signal events that might be of
>     interest.

Etc.

In the context of another simulator, we developed a different
technique, which would be quite general and might be of interest
for QEMU.  We communicate with the simulator via a "fake" *device*,
mapped into user-mode memory using an mmap call.  If someone
devised and coded such a device, then it could be used from any
guest.

Regards -- Eliot Moss

Re: [Qemu-devel] memory trace with qemu

[malc]

On Mon, 26 Jul 2010, Eliot Moss wrote:

> On 7/26/2010 6:20 AM, Llu?s wrote:
> > Eduardo Cruz writes:
> > 
> > > Thanks for your awnsers. Stean, after I find the right place to capture
> > > the
> > > reads and writes I'll definitely try your trace tool.
> > 
> > > Until now, this is what i found:
> > 
> > > I am using the x86-64 target, and I know that, for instance, lots of reads
> > > pass here:
> > 
> > > target-i386/translate.c   gen_op_ld_T1_A0()
> > 
> > Ok, I've seen at least 3 people working on this lately.
> > 
> > Some time ago I wrote a message proposing two sets of modifications for
> > qemu, in
> > order to allow the analysis of guest code (like feeding traces to an
> > architecture simulator).
> > 
> > What I proposed is based on two different functionalities:
> > 
> > 1) backdoor: a mechanism for the guest to communicate with qemu, such that
> >     tracing can be started, stopped, etc.
> > 
> >     My current approach is to decode an instruction that is deemed invalid
> > by the
> >     target ISA according to the manual.
> > 
> >     This is only implemented for x86 right now, but it is trivial to
> > implement on
> >     other architectures as long as there are unused opcodes.
> > 
> > 2) instrumentation: a set of generic macros that signal events that might be
> > of
> >     interest.
> 
> Etc.
> 
> In the context of another simulator, we developed a different
> technique, which would be quite general and might be of interest
> for QEMU.  We communicate with the simulator via a "fake" *device*,
> mapped into user-mode memory using an mmap call.  If someone
> devised and coded such a device, then it could be used from any
> guest.

I'v done something similar a while ago:
http://repo.or.cz/w/qemu/malc.git/shortlog/refs/heads/wctpci

-- 
mailto:av1474@comtv.ru

Re: [Qemu-devel] memory trace with qemu

[Lluís]

malc  writes:
>> > 2) instrumentation: a set of generic macros that signal events that might be
>> > of
>> >     interest.
>> 
>> Etc.
>> 
>> In the context of another simulator, we developed a different
>> technique, which would be quite general and might be of interest
>> for QEMU.  We communicate with the simulator via a "fake" *device*,
>> mapped into user-mode memory using an mmap call.  If someone
>> devised and coded such a device, then it could be used from any
>> guest.

> I'v done something similar a while ago:
> http://repo.or.cz/w/qemu/malc.git/shortlog/refs/heads/wctpci

Ah, although somewhat heavier, this is much more clever and generic :)

My only concern is on the need of running as root to mmap /dev/mem, but I could
write a dumb character device that can be mmaped by everybody.

Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: [Qemu-devel] memory trace with qemu

[Jun Koi]

On Mon, Jul 26, 2010 at 8:16 PM, malc <av1474@comtv.ru> wrote:
> On Mon, 26 Jul 2010, Eliot Moss wrote:
>
>> On 7/26/2010 6:20 AM, Llu?s wrote:
>> > Eduardo Cruz writes:
>> >
>> > > Thanks for your awnsers. Stean, after I find the right place to capture
>> > > the
>> > > reads and writes I'll definitely try your trace tool.
>> >
>> > > Until now, this is what i found:
>> >
>> > > I am using the x86-64 target, and I know that, for instance, lots of reads
>> > > pass here:
>> >
>> > > target-i386/translate.c   gen_op_ld_T1_A0()
>> >
>> > Ok, I've seen at least 3 people working on this lately.
>> >
>> > Some time ago I wrote a message proposing two sets of modifications for
>> > qemu, in
>> > order to allow the analysis of guest code (like feeding traces to an
>> > architecture simulator).
>> >
>> > What I proposed is based on two different functionalities:
>> >
>> > 1) backdoor: a mechanism for the guest to communicate with qemu, such that
>> >     tracing can be started, stopped, etc.
>> >
>> >     My current approach is to decode an instruction that is deemed invalid
>> > by the
>> >     target ISA according to the manual.
>> >
>> >     This is only implemented for x86 right now, but it is trivial to
>> > implement on
>> >     other architectures as long as there are unused opcodes.
>> >
>> > 2) instrumentation: a set of generic macros that signal events that might be
>> > of
>> >     interest.
>>
>> Etc.
>>
>> In the context of another simulator, we developed a different
>> technique, which would be quite general and might be of interest
>> for QEMU.  We communicate with the simulator via a "fake" *device*,
>> mapped into user-mode memory using an mmap call.  If someone
>> devised and coded such a device, then it could be used from any
>> guest.
>
> I'v done something similar a while ago:
> http://repo.or.cz/w/qemu/malc.git/shortlog/refs/heads/wctpci

in this repo tree, which part is your new code?

i looked at the git log, and tried to search your name, but none looks related.

thanks,
Jun

Re: [Qemu-devel] memory trace with qemu

[malc]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 988 bytes --]

On Tue, 27 Jul 2010, Jun Koi wrote:

> On Mon, Jul 26, 2010 at 8:16 PM, malc <av1474@comtv.ru> wrote:
> > On Mon, 26 Jul 2010, Eliot Moss wrote:
> >
> >> On 7/26/2010 6:20 AM, Llu?s wrote:
> >> > Eduardo Cruz writes:
[..snip..]

> >>
> >> In the context of another simulator, we developed a different
> >> technique, which would be quite general and might be of interest
> >> for QEMU.  We communicate with the simulator via a "fake" *device*,
> >> mapped into user-mode memory using an mmap call.  If someone
> >> devised and coded such a device, then it could be used from any
> >> guest.
> >
> > I'v done something similar a while ago:
> > http://repo.or.cz/w/qemu/malc.git/shortlog/refs/heads/wctpci
> 
> in this repo tree, which part is your new code?
> 
> i looked at the git log, and tried to search your name, but none looks related.

First five commits clearly have my nick in the second column, first
three commits are directly related to the branch.

-- 
mailto:av1474@comtv.ru

Re: [Qemu-devel] memory trace with qemu

[Eduardo Cruz]

Ok, I succeeded in adding the process tracker hack into qemu and
linux, and also in keeping track of the memory access (thanks Yufei
Chen for the tip).
I also simulated a cycle counter by counting the number of
instructions executed, which is equivalent to a cpu with cpi equals to
1.
But, as I am studying parallel programs behavior, I get very strange
results due to the fact that qemu executes thousands of instructions
of a cpu before moving to another cpu in smp systems.
I tried to limit this number, but it makes qemu very slow or even crash.
To achieve this, I added an helper that generates a timer event after
every X instructions executed.
I don't know If I am doing it in a wrong way.

Any suggestions?

2010/7/27 malc <av1474@comtv.ru>
>
> On Tue, 27 Jul 2010, Jun Koi wrote:
>
> > On Mon, Jul 26, 2010 at 8:16 PM, malc <av1474@comtv.ru> wrote:
> > > On Mon, 26 Jul 2010, Eliot Moss wrote:
> > >
> > >> On 7/26/2010 6:20 AM, Llu?s wrote:
> > >> > Eduardo Cruz writes:
> [..snip..]
>
> > >>
> > >> In the context of another simulator, we developed a different
> > >> technique, which would be quite general and might be of interest
> > >> for QEMU.  We communicate with the simulator via a "fake" *device*,
> > >> mapped into user-mode memory using an mmap call.  If someone
> > >> devised and coded such a device, then it could be used from any
> > >> guest.
> > >
> > > I'v done something similar a while ago:
> > > http://repo.or.cz/w/qemu/malc.git/shortlog/refs/heads/wctpci
> >
> > in this repo tree, which part is your new code?
> >
> > i looked at the git log, and tried to search your name, but none looks related.
>
> First five commits clearly have my nick in the second column, first
> three commits are directly related to the branch.
>
> --
> mailto:av1474@comtv.ru


--
Eduardo Henrique Molina da Cruz
MSc student
Parallel and Distributed Processing Group
Federal University of Rio Grande do Sul (UFRGS)

Re: [Qemu-devel] memory trace with qemu

[Yufei Chen]

On Mon, Jul 26, 2010 at 6:20 PM, Lluís <xscript@gmx.net> wrote:
> Eduardo Cruz writes:
>
>> Thanks for your awnsers. Stean, after I find the right place to capture the
>> reads and writes I'll definitely try your trace tool.
>
>> Until now, this is what i found:
>
>> I am using the x86-64 target, and I know that, for instance, lots of reads
>> pass here:
>
>> target-i386/translate.c   gen_op_ld_T1_A0()
>
> Ok, I've seen at least 3 people working on this lately.
>
> Some time ago I wrote a message proposing two sets of modifications for qemu, in
> order to allow the analysis of guest code (like feeding traces to an
> architecture simulator).
>
> What I proposed is based on two different functionalities:
>
> 1) backdoor: a mechanism for the guest to communicate with qemu, such that
>   tracing can be started, stopped, etc.
>
>   My current approach is to decode an instruction that is deemed invalid by the
>   target ISA according to the manual.
>
>   This is only implemented for x86 right now, but it is trivial to implement on
>   other architectures as long as there are unused opcodes.
>
> 2) instrumentation: a set of generic macros that signal events that might be of
>   interest.
>
>   The idea is to have the very same macros on all targets, such that the user
>   can just define what to do when these are called (like using the qemu tracing
>   mechanism), without bothering about the target architecture.
>
>   These include memory accesses, instruction fetch information (address, size,
>   used registers, defined registers and a generic opcode type), etc.
>   I started implementing it for x86, but the ISA is a hellish nightmare (no
>   news), and got sidetracked by other pending work.
>
>   The main idea is that instrumentation can be dynamically and selectively
>   activated/deactivated (thus the backdoor mechanism), and it is generated as
>   extra TCG code when active.

This is also what we are working on recently. We want to combine this
two functionalities to provide "smart watch pointer".

For example, the user can specify what memory location to be watched
(using backdoor). When the memory is accessed, it will trigger a
function in the underlying qemu to be called (using instrumentation).
With this mechanism, we can record memory access interleaving, guest
back trace and maybe other information that maybe useful for debugging
or other purposes.

>
> My intent was to spin up some discussion on which macros to provide, as well as
> the most performant mechanism for for the dynamic (de)activation of these.
>
> So, the question is, again, if the qemu community is willing to provide this, or
> should we maintain it as a separate set of patches.
>
> As I told Jun Koi, I'll try to set up a git repository at work for others to
> have access to the modifications and discuss the alternatives.
>
>
> Oh! BTW, my current prototype uses code like this on target-x86/translate.c:
>
>    #define tcg_gen_qemu_ld8u(arg, addr, mem_index)          \
>        do {                                                 \
>            INSTR_MEM(addr, 8);                              \
>            (tcg_gen_qemu_ld8u)(arg, addr, mem_index);       \
>        } while (0)
>
> Of course, this doesn't take into account DMA memory accesses and accesses
> produced by helper code.
>
>
> Lluís
>
> --
>  "And it's much the same thing with knowledge, for whenever you learn
>  something new, the whole world becomes that much richer."
>  -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
>  Tollbooth
>



-- 
Best regards,
Chen Yufei

Re: [Qemu-devel] memory trace with qemu

[Eduardo Cruz]

In the modification:
 Oh! BTW, my current prototype uses code like this on target-x86/translate.c:
>
>    #define tcg_gen_qemu_ld8u(arg, addr, mem_index)          \
>        do {                                                 \
>            INSTR_MEM(addr, 8);                              \
>            (tcg_gen_qemu_ld8u)(arg, addr, mem_index);       \
>        } while (0)

As far as I understand, at this point, the code is still beig
translated, so we still dont know at this point the virtual address,
because the address will be computed when the generated code executes.
Am I correct? or am I completely lost?

I think qemu should support the memory trace, and I'll try to do it.
I've already managed to hack the linux kernel to make it easy to
obtain the pid of the current process running on the cpu. If we can
make qemu generate the trace, we can make it only record the memory
access of specific process. I already do this using simics, but it is
extremely slow.

2010/7/26 Yufei Chen <cyfdecyf@gmail.com>
>
> On Mon, Jul 26, 2010 at 6:20 PM, Lluís <xscript@gmx.net> wrote:
> > Eduardo Cruz writes:
> >
> >> Thanks for your awnsers. Stean, after I find the right place to capture the
> >> reads and writes I'll definitely try your trace tool.
> >
> >> Until now, this is what i found:
> >
> >> I am using the x86-64 target, and I know that, for instance, lots of reads
> >> pass here:
> >
> >> target-i386/translate.c   gen_op_ld_T1_A0()
> >
> > Ok, I've seen at least 3 people working on this lately.
> >
> > Some time ago I wrote a message proposing two sets of modifications for qemu, in
> > order to allow the analysis of guest code (like feeding traces to an
> > architecture simulator).
> >
> > What I proposed is based on two different functionalities:
> >
> > 1) backdoor: a mechanism for the guest to communicate with qemu, such that
> >   tracing can be started, stopped, etc.
> >
> >   My current approach is to decode an instruction that is deemed invalid by the
> >   target ISA according to the manual.
> >
> >   This is only implemented for x86 right now, but it is trivial to implement on
> >   other architectures as long as there are unused opcodes.
> >
> > 2) instrumentation: a set of generic macros that signal events that might be of
> >   interest.
> >
> >   The idea is to have the very same macros on all targets, such that the user
> >   can just define what to do when these are called (like using the qemu tracing
> >   mechanism), without bothering about the target architecture.
> >
> >   These include memory accesses, instruction fetch information (address, size,
> >   used registers, defined registers and a generic opcode type), etc.
> >   I started implementing it for x86, but the ISA is a hellish nightmare (no
> >   news), and got sidetracked by other pending work.
> >
> >   The main idea is that instrumentation can be dynamically and selectively
> >   activated/deactivated (thus the backdoor mechanism), and it is generated as
> >   extra TCG code when active.
>
> This is also what we are working on recently. We want to combine this
> two functionalities to provide "smart watch pointer".
>
> For example, the user can specify what memory location to be watched
> (using backdoor). When the memory is accessed, it will trigger a
> function in the underlying qemu to be called (using instrumentation).
> With this mechanism, we can record memory access interleaving, guest
> back trace and maybe other information that maybe useful for debugging
> or other purposes.
>
> >
> > My intent was to spin up some discussion on which macros to provide, as well as
> > the most performant mechanism for for the dynamic (de)activation of these.
> >
> > So, the question is, again, if the qemu community is willing to provide this, or
> > should we maintain it as a separate set of patches.
> >
> > As I told Jun Koi, I'll try to set up a git repository at work for others to
> > have access to the modifications and discuss the alternatives.
> >
> >
> > Oh! BTW, my current prototype uses code like this on target-x86/translate.c:
> >
> >    #define tcg_gen_qemu_ld8u(arg, addr, mem_index)          \
> >        do {                                                 \
> >            INSTR_MEM(addr, 8);                              \
> >            (tcg_gen_qemu_ld8u)(arg, addr, mem_index);       \
> >        } while (0)
> >
> > Of course, this doesn't take into account DMA memory accesses and accesses
> > produced by helper code.
> >
> >
> > Lluís
> >
> > --
> >  "And it's much the same thing with knowledge, for whenever you learn
> >  something new, the whole world becomes that much richer."
> >  -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
> >  Tollbooth
> >
>
>
>
> --
> Best regards,
> Chen Yufei



--
Eduardo Henrique Molina da Cruz
MSc student
Parallel and Distributed Processing Group
Federal University of Rio Grande do Sul (UFRGS)

Re: [Qemu-devel] memory trace with qemu

[Lluís]

Eduardo Cruz writes:

> In the modification:
>  Oh! BTW, my current prototype uses code like this on target-x86/translate.c:
>> 
>>    #define tcg_gen_qemu_ld8u(arg, addr, mem_index)          \
>>        do {                                                 \
>>            INSTR_MEM(addr, 8);                              \
>>            (tcg_gen_qemu_ld8u)(arg, addr, mem_index);       \
>>        } while (0)

> As far as I understand, at this point, the code is still beig
> translated, so we still dont know at this point the virtual address,
> because the address will be computed when the generated code executes.
> Am I correct? or am I completely lost?

I define INSTR_MEM macro into a generation of a helper call, which will receive
the run-time computed address.


> I think qemu should support the memory trace, and I'll try to do it.
> I've already managed to hack the linux kernel to make it easy to
> obtain the pid of the current process running on the cpu. If we can
> make qemu generate the trace, we can make it only record the memory
> access of specific process. I already do this using simics, but it is
> extremely slow.

I have already done all this with bochs, and I'm trying to port it into qemu for
speed :)

The trick is extremely easy, as long as you don't want to differentiate between
pthreads. You just have to track modifications of the cr3 in x86. In fact, I was
planning on providing a generic INSTR_ASPACE macro, that would signal a change
of address space.

If you'd also like to track pthreads and the like, you'll need some sort of
backdoor communication channel.


Right now I'm stuck trying to let the user define a directory from where a
library will be compiled for each target, such that not only code helpers and
the like can be defined, but the user will also be available to link any other
piece of external software (e.g., I have a separate full-fledged conditional
tracing infrastructure that I'd like to call from the instrumentation macros).

Just let me finish the this bit of Makefile black magic and I'll push the
repository into a public place.

Lluis


-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: [Qemu-devel] memory trace with qemu

[Eduardo Cruz]

> I define INSTR_MEM macro into a generation of a helper call, which will receive
> the run-time computed address.

Could you share how did you do this?

> If you'd also like to track pthreads and the like, you'll need some sort of
> backdoor communication channel.

Yes, I need to distinguish between threads of the same process, but I
already solved this by hacking the linux context switcher procedure.

2010/7/26 Lluís <xscript@gmx.net>:
> Eduardo Cruz writes:
>
>> In the modification:
>>  Oh! BTW, my current prototype uses code like this on target-x86/translate.c:
>>>
>>>    #define tcg_gen_qemu_ld8u(arg, addr, mem_index)          \
>>>        do {                                                 \
>>>            INSTR_MEM(addr, 8);                              \
>>>            (tcg_gen_qemu_ld8u)(arg, addr, mem_index);       \
>>>        } while (0)
>
>> As far as I understand, at this point, the code is still beig
>> translated, so we still dont know at this point the virtual address,
>> because the address will be computed when the generated code executes.
>> Am I correct? or am I completely lost?
>
> I define INSTR_MEM macro into a generation of a helper call, which will receive
> the run-time computed address.
>
>
>> I think qemu should support the memory trace, and I'll try to do it.
>> I've already managed to hack the linux kernel to make it easy to
>> obtain the pid of the current process running on the cpu. If we can
>> make qemu generate the trace, we can make it only record the memory
>> access of specific process. I already do this using simics, but it is
>> extremely slow.
>
> I have already done all this with bochs, and I'm trying to port it into qemu for
> speed :)
>
> The trick is extremely easy, as long as you don't want to differentiate between
> pthreads. You just have to track modifications of the cr3 in x86. In fact, I was
> planning on providing a generic INSTR_ASPACE macro, that would signal a change
> of address space.
>
> If you'd also like to track pthreads and the like, you'll need some sort of
> backdoor communication channel.
>
>
> Right now I'm stuck trying to let the user define a directory from where a
> library will be compiled for each target, such that not only code helpers and
> the like can be defined, but the user will also be available to link any other
> piece of external software (e.g., I have a separate full-fledged conditional
> tracing infrastructure that I'd like to call from the instrumentation macros).
>
> Just let me finish the this bit of Makefile black magic and I'll push the
> repository into a public place.
>
> Lluis
>
>
> --
>  "And it's much the same thing with knowledge, for whenever you learn
>  something new, the whole world becomes that much richer."
>  -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
>  Tollbooth
>



-- 
Eduardo Henrique Molina da Cruz
MSc student
Parallel and Distributed Processing Group
Federal University of Rio Grande do Sul (UFRGS)