[Qemu-devel] Qemu Trace

[Qemu-devel] Qemu Trace

[Nesrine Zouari]

Hello,

I am a computer engineering student and I am actually working on my
graduation project at Lauterbach company. The project is about Qemu Trace
and as a future I would like to contribute this work to the main line.

My project is divided into two parts:

1/ Collecting the Guest trace data : The trace solution should be able to
provide:

            a/ Instruction flow Trace

            b/ Memory read/write access

            c/ Time Stamps.

            d/ For tracing rich operating systems that are using MMU, we
additionally need to trace the task switches.

2/ Sending the collected data to a third party tool for analysis.

My question is about the first part. I would like to know, which trace
backend that better fit my use case.


Regards,

-----------------------------------------------------------------------------------------

Nesrine ZOUARI
Computer Engineering Student
Department of Computer Engineering and Applied Mathematics
National Engineering School of Sfax (ENIS)
University of Sfax-Tunisia
Tel: +216 52 620 475

Re: [Qemu-devel] Qemu Trace

[Dongli Zhang]

Hi Nesrine,

On 02/01/2018 11:30 PM, Nesrine Zouari wrote:
> Hello,
> 
> I am a computer engineering student and I am actually working on my
> graduation project at Lauterbach company. The project is about Qemu Trace
> and as a future I would like to contribute this work to the main line.
> 
> My project is divided into two parts:
> 
> 1/ Collecting the Guest trace data : The trace solution should be able to
> provide:
> 
>             a/ Instruction flow Trace
> 
>             b/ Memory read/write access

About mem read/write, there is a project panda:

https://github.com/panda-re/panda

> 
>             c/ Time Stamps.
> 
>             d/ For tracing rich operating systems that are using MMU, we
> additionally need to trace the task switches.
> 
> 2/ Sending the collected data to a third party tool for analysis.
> 
> My question is about the first part. I would like to know, which trace
> backend that better fit my use case.
> 
> 
> Regards,
> 
> -----------------------------------------------------------------------------------------
> 
> Nesrine ZOUARI
> Computer Engineering Student
> Department of Computer Engineering and Applied Mathematics
> National Engineering School of Sfax (ENIS)
> University of Sfax-Tunisia
> Tel: +216 52 620 475
> 

Dongli Zhang

Re: [Qemu-devel] Qemu Trace

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2126 bytes --]

On Thu, Feb 01, 2018 at 04:30:10PM +0100, Nesrine Zouari wrote:
> I am a computer engineering student and I am actually working on my
> graduation project at Lauterbach company. The project is about Qemu Trace
> and as a future I would like to contribute this work to the main line.
> 
> My project is divided into two parts:
> 
> 1/ Collecting the Guest trace data : The trace solution should be able to
> provide:
> 
>             a/ Instruction flow Trace
> 
>             b/ Memory read/write access
> 
>             c/ Time Stamps.
> 
>             d/ For tracing rich operating systems that are using MMU, we
> additionally need to trace the task switches.

Lluìs has done the most instrumentation work in qemu.git and can explain
the current status.

The focus in QEMU is more on functional simulation than on low-level
instrumentation.  Therefore the instrumentation facilities aren't very
rich.  Code changes will be required to get the information you need.
In order to be suitable for upstream they should not be too invasive or
impact performance significantly.

Which CPU architecture are you targeting?

> 2/ Sending the collected data to a third party tool for analysis.
> 
> My question is about the first part. I would like to know, which trace
> backend that better fit my use case.

LTTng UST has the highest performance tracing interface.  It uses shared
memory to efficiently export trace data to a collector or analysis
process.

It is probably not necessary to invent your own tracer or interface for
capturing trace data.  I suggest looking into LTTng UST and trying it
out.

The basic idea would be:

1. Add missing trace events to QEMU
2. Build with ./configure --enable-trace-backend=ust && make
3. Use LTTng tools or write your own collector using the LTTng libraries
4. Enable the trace events that you need for instruction flow, memory
   access, and task switching.

The QEMU code changes involved would be changes to trace-events and
placing those trace events into TCG and/or memory API code to record the
necessary information.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Qemu Trace

[Nesrine Zouari]

Hello Mr. Stefan,

Thank you for your response. To answer your question : the trace solution
should be architecture independent.This is the best for us. We aim to test
it at least for ARM/ARM64 , x86 and x64 architectures. But even if  there
will be some differences , we can accept it.

Regards,

On Fri, Feb 2, 2018 at 11:08 AM, Stefan Hajnoczi <stefanha@redhat.com>
wrote:

> On Thu, Feb 01, 2018 at 04:30:10PM +0100, Nesrine Zouari wrote:
> > I am a computer engineering student and I am actually working on my
> > graduation project at Lauterbach company. The project is about Qemu Trace
> > and as a future I would like to contribute this work to the main line.
> >
> > My project is divided into two parts:
> >
> > 1/ Collecting the Guest trace data : The trace solution should be able to
> > provide:
> >
> >             a/ Instruction flow Trace
> >
> >             b/ Memory read/write access
> >
> >             c/ Time Stamps.
> >
> >             d/ For tracing rich operating systems that are using MMU, we
> > additionally need to trace the task switches.
>
> Lluìs has done the most instrumentation work in qemu.git and can explain
> the current status.
>
> The focus in QEMU is more on functional simulation than on low-level
> instrumentation.  Therefore the instrumentation facilities aren't very
> rich.  Code changes will be required to get the information you need.
> In order to be suitable for upstream they should not be too invasive or
> impact performance significantly.
>
> Which CPU architecture are you targeting?
>
> > 2/ Sending the collected data to a third party tool for analysis.
> >
> > My question is about the first part. I would like to know, which trace
> > backend that better fit my use case.
>
> LTTng UST has the highest performance tracing interface.  It uses shared
> memory to efficiently export trace data to a collector or analysis
> process.
>
> It is probably not necessary to invent your own tracer or interface for
> capturing trace data.  I suggest looking into LTTng UST and trying it
> out.
>
> The basic idea would be:
>
> 1. Add missing trace events to QEMU
> 2. Build with ./configure --enable-trace-backend=ust && make
> 3. Use LTTng tools or write your own collector using the LTTng libraries
> 4. Enable the trace events that you need for instruction flow, memory
>    access, and task switching.
>
> The QEMU code changes involved would be changes to trace-events and
> placing those trace events into TCG and/or memory API code to record the
> necessary information.
>
> Stefan
>



-- 
-----------------------------------------------------------------------------------------

Nesrine ZOUARI
Computer Engineering Student
Department of Computer Engineering and Applied Mathematics
National Engineering School of Sfax (ENIS)
University of Sfax-Tunisia
Tel: +216 52 620 475

Re: [Qemu-devel] Qemu Trace

[Peter Maydell]

On 2 February 2018 at 10:08, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> On Thu, Feb 01, 2018 at 04:30:10PM +0100, Nesrine Zouari wrote:
>> I am a computer engineering student and I am actually working on my
>> graduation project at Lauterbach company. The project is about Qemu Trace
>> and as a future I would like to contribute this work to the main line.
>>
>> My project is divided into two parts:
>>
>> 1/ Collecting the Guest trace data : The trace solution should be able to
>> provide:
>>
>>             a/ Instruction flow Trace
>>
>>             b/ Memory read/write access
>>
>>             c/ Time Stamps.
>>
>>             d/ For tracing rich operating systems that are using MMU, we
>> additionally need to trace the task switches.
>
> Lluìs has done the most instrumentation work in qemu.git and can explain
> the current status.

I think at the moment the status is that we're still discussing
what the trace plugin API should be... there are some mailing
list threads on the subject from I think last year some time.

thanks
-- PMM

Re: [Qemu-devel] Qemu Trace

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1986 bytes --]

On Fri, Feb 02, 2018 at 03:53:45PM +0000, Peter Maydell wrote:
> On 2 February 2018 at 10:08, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > On Thu, Feb 01, 2018 at 04:30:10PM +0100, Nesrine Zouari wrote:
> >> I am a computer engineering student and I am actually working on my
> >> graduation project at Lauterbach company. The project is about Qemu Trace
> >> and as a future I would like to contribute this work to the main line.
> >>
> >> My project is divided into two parts:
> >>
> >> 1/ Collecting the Guest trace data : The trace solution should be able to
> >> provide:
> >>
> >>             a/ Instruction flow Trace
> >>
> >>             b/ Memory read/write access
> >>
> >>             c/ Time Stamps.
> >>
> >>             d/ For tracing rich operating systems that are using MMU, we
> >> additionally need to trace the task switches.
> >
> > Lluìs has done the most instrumentation work in qemu.git and can explain
> > the current status.
> 
> I think at the moment the status is that we're still discussing
> what the trace plugin API should be... there are some mailing

s/trace plugin API/instrumentation plugin API/

> list threads on the subject from I think last year some time.

The point of the instrumentation plugin API is for online analysis
(stuff that cannot be post-processed offline) with the ability for the
plugin to control QEMU (e.g. affect translation during a run).  That is
not necessary for Nesrine's use case.

I see two main requirements in Nesrine's use case:

1. Instruction flow, memory access, and task switch trace.  This doesn't
exist today and requires adding new trace events.

2. An external interface for processing traces.  There seems to be a
requirement for doing it online and not by post-processing trace files.
Therefore I suggested looking into LTTng UST, which has an efficient
interface and libraries for tracing processes.  I think no QEMU changes
are necessary here.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Qemu Trace

[Peter Maydell]

On 5 February 2018 at 15:51, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> The point of the instrumentation plugin API is for online analysis
> (stuff that cannot be post-processed offline) with the ability for the
> plugin to control QEMU (e.g. affect translation during a run).  That is
> not necessary for Nesrine's use case.

...but it hugely overlaps, because the most interesting
reason for having a trace plugin is to be able to do things
like tracing instruction flow, memory access and so on.

> I see two main requirements in Nesrine's use case:
>
> 1. Instruction flow, memory access, and task switch trace.  This doesn't
> exist today and requires adding new trace events.
>
> 2. An external interface for processing traces.  There seems to be a
> requirement for doing it online and not by post-processing trace files.
> Therefore I suggested looking into LTTng UST, which has an efficient
> interface and libraries for tracing processes.  I think no QEMU changes
> are necessary here.

...and "online processing of traces" is pretty much exactly
what an instrumentation plugin API is for.

It would also let you put the logic of "how do we know what a
guest OS task switch is anyway" somewhere other than QEMU,
which is nice because that's very guest-OS specific.

thanks
-- PMM

Re: [Qemu-devel] Qemu Trace

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2731 bytes --]

On Mon, Feb 05, 2018 at 03:55:01PM +0000, Peter Maydell wrote:
> On 5 February 2018 at 15:51, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > The point of the instrumentation plugin API is for online analysis
> > (stuff that cannot be post-processed offline) with the ability for the
> > plugin to control QEMU (e.g. affect translation during a run).  That is
> > not necessary for Nesrine's use case.
> 
> ...but it hugely overlaps, because the most interesting
> reason for having a trace plugin is to be able to do things
> like tracing instruction flow, memory access and so on.
> 
> > I see two main requirements in Nesrine's use case:
> >
> > 1. Instruction flow, memory access, and task switch trace.  This doesn't
> > exist today and requires adding new trace events.
> >
> > 2. An external interface for processing traces.  There seems to be a
> > requirement for doing it online and not by post-processing trace files.
> > Therefore I suggested looking into LTTng UST, which has an efficient
> > interface and libraries for tracing processes.  I think no QEMU changes
> > are necessary here.
> 
> ...and "online processing of traces" is pretty much exactly
> what an instrumentation plugin API is for.

There are two cases for online processing:

1. Synchronous - this is what the instrumentation plugins will be able
   to do.  It means that QEMU waits until the event has been processed,
   and the plugin is therefore able to control QEMU.

2. Asynchronous - this is what LTTng UST does.  It means that the
   process analyzing the trace does not hold up QEMU, but it cannot
   control QEMU without some delay between trace events firing and
   reacting to them.  I think Nesrine's use case is this one.

#2 is useful for "real-time" graphs, for example.  That is still
"oneline" but it doesn't require instrumentation plugins.

Given that the instrumentation plugin API is still in early development,
it would delay Nesrine's work and I recommend against depending on it.

> It would also let you put the logic of "how do we know what a
> guest OS task switch is anyway" somewhere other than QEMU,
> which is nice because that's very guest-OS specific.

How do you envision that instrumentation plugins will monitor guest OS
task switches if QEMU does not know about them?

Without instrumentation plugins I would add a guest_x86_cr3_write trace
event, for example.  Then the trace consumer can interpret that event as
a task switch if it knows the guest OS always writes to the cr3 register
on a task switch.  No instrumentation plugins are necessary in order to
do this and it doesn't hardcode knowledge of guest OS task switching
into QEMU either.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Qemu Trace

[Peter Maydell]

On 6 February 2018 at 09:17, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> On Mon, Feb 05, 2018 at 03:55:01PM +0000, Peter Maydell wrote:
>> ...and "online processing of traces" is pretty much exactly
>> what an instrumentation plugin API is for.
>
> There are two cases for online processing:
>
> 1. Synchronous - this is what the instrumentation plugins will be able
>    to do.  It means that QEMU waits until the event has been processed,
>    and the plugin is therefore able to control QEMU.
>
> 2. Asynchronous - this is what LTTng UST does.  It means that the
>    process analyzing the trace does not hold up QEMU, but it cannot
>    control QEMU without some delay between trace events firing and
>    reacting to them.  I think Nesrine's use case is this one.
>
> #2 is useful for "real-time" graphs, for example.  That is still
> "oneline" but it doesn't require instrumentation plugins.
>
> Given that the instrumentation plugin API is still in early development,
> it would delay Nesrine's work and I recommend against depending on it.

I don't really want us to end up with two separate ways of doing
basically the same thing that have been implemented entirely
independently.

A lot of the problems you need to solve (like "how do we
configurably add tracing of things like guest register changes
into the generated code in a maintainable and efficient way")
are the same both ways.

>> It would also let you put the logic of "how do we know what a
>> guest OS task switch is anyway" somewhere other than QEMU,
>> which is nice because that's very guest-OS specific.
>
> How do you envision that instrumentation plugins will monitor guest OS
> task switches if QEMU does not know about them?

The plugin can register for information like changes of the
relevant system registers (and could potentially do cleverer
things like looking at guest in-memory task data structures).
QEMU doesn't need to know (and shouldn't know) that x86 CR3 or
Arm CONTEXTIDR are how you identify a guest OS task switch.

> Without instrumentation plugins I would add a guest_x86_cr3_write trace
> event, for example.  Then the trace consumer can interpret that event as
> a task switch if it knows the guest OS always writes to the cr3 register
> on a task switch.  No instrumentation plugins are necessary in order to
> do this and it doesn't hardcode knowledge of guest OS task switching
> into QEMU either.

This only works if you have a trace event for the register change,
which we don't.

thanks
-- PMM

Re: [Qemu-devel] Qemu Trace

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3301 bytes --]

On Tue, Feb 06, 2018 at 10:55:57AM +0000, Peter Maydell wrote:
> On 6 February 2018 at 09:17, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > On Mon, Feb 05, 2018 at 03:55:01PM +0000, Peter Maydell wrote:
> >> ...and "online processing of traces" is pretty much exactly
> >> what an instrumentation plugin API is for.
> >
> > There are two cases for online processing:
> >
> > 1. Synchronous - this is what the instrumentation plugins will be able
> >    to do.  It means that QEMU waits until the event has been processed,
> >    and the plugin is therefore able to control QEMU.
> >
> > 2. Asynchronous - this is what LTTng UST does.  It means that the
> >    process analyzing the trace does not hold up QEMU, but it cannot
> >    control QEMU without some delay between trace events firing and
> >    reacting to them.  I think Nesrine's use case is this one.
> >
> > #2 is useful for "real-time" graphs, for example.  That is still
> > "oneline" but it doesn't require instrumentation plugins.
> >
> > Given that the instrumentation plugin API is still in early development,
> > it would delay Nesrine's work and I recommend against depending on it.
> 
> I don't really want us to end up with two separate ways of doing
> basically the same thing that have been implemented entirely
> independently.

I agree it shouldn't be duplicated.  The instrumentation plugin
infrastructure can use tracing.

What I'm not convinced of is that Nesrine's use case requires a plugin
that can react to events within QEMU.  It seems like it just needs to
export a stream of events to an external process - instrumentation
plugins aren't needed for that, just use LTTng UST.

> A lot of the problems you need to solve (like "how do we
> configurably add tracing of things like guest register changes
> into the generated code in a maintainable and efficient way")
> are the same both ways.

Tracing already has special "tcg" events, which can be planted in
generated code as well as triggered at translation time.  So I think
this is already solved?

> >> It would also let you put the logic of "how do we know what a
> >> guest OS task switch is anyway" somewhere other than QEMU,
> >> which is nice because that's very guest-OS specific.
> >
> > How do you envision that instrumentation plugins will monitor guest OS
> > task switches if QEMU does not know about them?
> 
> The plugin can register for information like changes of the
> relevant system registers (and could potentially do cleverer
> things like looking at guest in-memory task data structures).
> QEMU doesn't need to know (and shouldn't know) that x86 CR3 or
> Arm CONTEXTIDR are how you identify a guest OS task switch.
> 
> > Without instrumentation plugins I would add a guest_x86_cr3_write trace
> > event, for example.  Then the trace consumer can interpret that event as
> > a task switch if it knows the guest OS always writes to the cr3 register
> > on a task switch.  No instrumentation plugins are necessary in order to
> > do this and it doesn't hardcode knowledge of guest OS task switching
> > into QEMU either.
> 
> This only works if you have a trace event for the register change,
> which we don't.

Adding a trace event for cr3 updates looks easy.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]