[Qemu-devel] Tracing: outstanding tasks

[Qemu-devel] Tracing: outstanding tasks

[Stefan Hajnoczi]

Here are the outstanding tasks for QEMU tracing, which Prerna and I have
been working on.  Tracing aids debugging, profiling, and observing
execution via lightweight logging at key points in the code path.

The current prototype is available from the 'tracing' branch at:

http://repo.or.cz/w/qemu/stefanha.git/shortlog/refs/heads/tracing

This email is both to help Prerna and me focus our efforts, as well as a
roadmap for the QEMU community to discuss and comment on.

Here are the outstanding tasks for a tracing patchset that can be
proposed for merge:

1. Integration with QMP

Owner: Prerna

Currently the trace commands are available from the monitor but a QMP interface
is needed.

2. More tracepoints need to be added for instrumenting other qemu components
such as virtio drivers, etc.

Owner: ?

QEMU must come with a useful set of trace events that allows people to get up
and running quickly.  Focus areas include:

 * guest device emulation
 * host devices
 * lifecycle and runloop
 * memory management
 * live migration

3. Documentation

Owner: Stefan

User documentation that explains how to collect traces and add new trace
events.  I have committed documentation to the tracing branch here:

http://repo.or.cz/w/qemu/stefanha.git/blob_plain/91fde34dc6bfe01af6d5e9265f6a81535d6add15:/docs/tracing.txt

4. Fix i386-linux-user build

Owner: Prerna

  LINK  i386-linux-user/qemu-i386
../simpletrace.o: In function `do_info_all_trace_events':
/home/stefanha/qemu/simpletrace.c:88: undefined reference to `monitor_printf'
../simpletrace.o: In function `do_info_trace':
/home/stefanha/qemu/simpletrace.c:77: undefined reference to `monitor_printf'

5. Out-of-line trace file write-out

Owner: Stefan

Trace buffers are written out to file synchronously.  The vcpu thread should
not be blocked so an async write-out mechanism is needed.

6. Trace file command

Owner: ?

Traces are written out to hardcoded /tmp/trace.log.  This must be configurable.
Tracing at startup time should still be possible so configuration needs to
happen early.

7. Binary trace format finalization

Owner: Stefan

We should leave room for extension.  I suggest partitioning the Event ID
namespace into normal events and special events.  The __trace_begin special
event is defined to contain the file format version and/or trace record size in
bytes as the first trace record in the file.

This way, post-processing tools can check the format of the binary trace file.

8. QMP/monitor command review

Owner: Prerna, Stefan

Are we happy to support the trace commands in the longer term (i.e. did we get
the commands and syntax right)?  Is input being validated properly?

Stefan

Re: [Qemu-devel] Tracing: outstanding tasks

[Prerna Saxena]

On 06/26/2010 01:36 PM, Stefan Hajnoczi wrote:
> Here are the outstanding tasks for QEMU tracing, which Prerna and I have
> been working on.  Tracing aids debugging, profiling, and observing
> execution via lightweight logging at key points in the code path.
>
> The current prototype is available from the 'tracing' branch at:
>
> http://repo.or.cz/w/qemu/stefanha.git/shortlog/refs/heads/tracing
>
> This email is both to help Prerna and me focus our efforts, as well as a
> roadmap for the QEMU community to discuss and comment on.
>
> Here are the outstanding tasks for a tracing patchset that can be
> proposed for merge:
>
> 1. Integration with QMP
>
> Owner: Prerna
>
> Currently the trace commands are available from the monitor but a QMP interface
> is needed.
>

Agree. I'd suggest we let this be a TODO till the trace infrastructure 
makes its way upstream.

> 2. More tracepoints need to be added for instrumenting other qemu components
> such as virtio drivers, etc.
>
> Owner: ?
>
> QEMU must come with a useful set of trace events that allows people to get up
> and running quickly.  Focus areas include:
>
>   * guest device emulation
>   * host devices
>   * lifecycle and runloop
>   * memory management
>   * live migration
>
> 3. Documentation
>
> Owner: Stefan
>
> User documentation that explains how to collect traces and add new trace
> events.  I have committed documentation to the tracing branch here:
>
> http://repo.or.cz/w/qemu/stefanha.git/blob_plain/91fde34dc6bfe01af6d5e9265f6a81535d6add15:/docs/tracing.txt
>
> 4. Fix i386-linux-user build
>
> Owner: Prerna
>
>    LINK  i386-linux-user/qemu-i386
> ../simpletrace.o: In function `do_info_all_trace_events':
> /home/stefanha/qemu/simpletrace.c:88: undefined reference to `monitor_printf'
> ../simpletrace.o: In function `do_info_trace':
> /home/stefanha/qemu/simpletrace.c:77: undefined reference to `monitor_printf'
>

I'm looking into this.

> 5. Out-of-line trace file write-out
>
> Owner: Stefan
>
> Trace buffers are written out to file synchronously.  The vcpu thread should
> not be blocked so an async write-out mechanism is needed.
>
> 6. Trace file command
>
> Owner: ?
>
> Traces are written out to hardcoded /tmp/trace.log.  This must be configurable.
> Tracing at startup time should still be possible so configuration needs to
> happen early.

Agree, it is a good-to-have feature.

>
> 7. Binary trace format finalization
>
> Owner: Stefan
>
> We should leave room for extension.  I suggest partitioning the Event ID
> namespace into normal events and special events.  The __trace_begin special
> event is defined to contain the file format version and/or trace record size in
> bytes as the first trace record in the file.
>
> This way, post-processing tools can check the format of the binary trace file.
>
> 8. QMP/monitor command review
>
> Owner: Prerna, Stefan
>
>

9. In its present format, the tracing infrastructure causes at least two 
function calls even for trace events that are disabled. Ideally, there 
should be minimal performance overhead for disabled trace events.
Investigating scope for further optimization.

Owner : Prerna

Thanks,
-- 
Prerna Saxena

Linux Technology Centre,
IBM Systems and Technology Lab,
Bangalore, India

Re: [Qemu-devel] Tracing: outstanding tasks

[Stefan Hajnoczi]

On Wed, Jun 30, 2010 at 11:20 AM, Prerna Saxena
<prerna@linux.vnet.ibm.com> wrote:
> On 06/26/2010 01:36 PM, Stefan Hajnoczi wrote:
>>
>> Here are the outstanding tasks for QEMU tracing, which Prerna and I have
>> been working on.  Tracing aids debugging, profiling, and observing
>> execution via lightweight logging at key points in the code path.
>>
>> The current prototype is available from the 'tracing' branch at:
>>
>> http://repo.or.cz/w/qemu/stefanha.git/shortlog/refs/heads/tracing
>>
>> This email is both to help Prerna and me focus our efforts, as well as a
>> roadmap for the QEMU community to discuss and comment on.
>>
>> Here are the outstanding tasks for a tracing patchset that can be
>> proposed for merge:
>>
>> 1. Integration with QMP
>>
>> Owner: Prerna
>>
>> Currently the trace commands are available from the monitor but a QMP
>> interface
>> is needed.
>>
>
> Agree. I'd suggest we let this be a TODO till the trace infrastructure makes
> its way upstream.
>
>> 2. More tracepoints need to be added for instrumenting other qemu
>> components
>> such as virtio drivers, etc.
>>
>> Owner: ?
>>
>> QEMU must come with a useful set of trace events that allows people to get
>> up
>> and running quickly.  Focus areas include:
>>
>>  * guest device emulation
>>  * host devices
>>  * lifecycle and runloop
>>  * memory management
>>  * live migration
>>
>> 3. Documentation
>>
>> Owner: Stefan
>>
>> User documentation that explains how to collect traces and add new trace
>> events.  I have committed documentation to the tracing branch here:
>>
>>
>> http://repo.or.cz/w/qemu/stefanha.git/blob_plain/91fde34dc6bfe01af6d5e9265f6a81535d6add15:/docs/tracing.txt
>>
>> 4. Fix i386-linux-user build
>>
>> Owner: Prerna
>>
>>   LINK  i386-linux-user/qemu-i386
>> ../simpletrace.o: In function `do_info_all_trace_events':
>> /home/stefanha/qemu/simpletrace.c:88: undefined reference to
>> `monitor_printf'
>> ../simpletrace.o: In function `do_info_trace':
>> /home/stefanha/qemu/simpletrace.c:77: undefined reference to
>> `monitor_printf'
>>
>
> I'm looking into this.
>
>> 5. Out-of-line trace file write-out
>>
>> Owner: Stefan
>>
>> Trace buffers are written out to file synchronously.  The vcpu thread
>> should
>> not be blocked so an async write-out mechanism is needed.
>>
>> 6. Trace file command
>>
>> Owner: ?
>>
>> Traces are written out to hardcoded /tmp/trace.log.  This must be
>> configurable.
>> Tracing at startup time should still be possible so configuration needs to
>> happen early.
>
> Agree, it is a good-to-have feature.
>
>>
>> 7. Binary trace format finalization
>>
>> Owner: Stefan
>>
>> We should leave room for extension.  I suggest partitioning the Event ID
>> namespace into normal events and special events.  The __trace_begin
>> special
>> event is defined to contain the file format version and/or trace record
>> size in
>> bytes as the first trace record in the file.
>>
>> This way, post-processing tools can check the format of the binary trace
>> file.
>>
>> 8. QMP/monitor command review
>>
>> Owner: Prerna, Stefan
>>
>>
>
> 9. In its present format, the tracing infrastructure causes at least two
> function calls even for trace events that are disabled. Ideally, there
> should be minimal performance overhead for disabled trace events.
> Investigating scope for further optimization.
>
> Owner : Prerna

Have you looked at the immediate value abstraction in LTTng?  I didn't
look very far, but I believe it abstracts how you enable/disable
breakpoints.  A simple immediate value implementation stores the state
in a variable like we do for the simple trace backend
(TraceEvent.state).  A "smarter" implementation is an unconditional
jump over the machine instructions that call into trace().  When the
state of a trace event is toggled, the jump instruction is patched.

That said, I think we should keep the "simple" trace backend simple as
long as possible.  Hopefully platform tracing mechanism like LTTng or
SystemTap will mature and become more widely supported so that we
don't need to re-invent all the hard parts of tracing.

Stefan

Re: [Qemu-devel] Tracing: outstanding tasks

[Ananth N Mavinakayanahalli]

On Wed, Jun 30, 2010 at 12:51:59PM +0100, Stefan Hajnoczi wrote:
> On Wed, Jun 30, 2010 at 11:20 AM, Prerna Saxena
> <prerna@linux.vnet.ibm.com> wrote:
> > On 06/26/2010 01:36 PM, Stefan Hajnoczi wrote:
> >>
> >> Here are the outstanding tasks for QEMU tracing, which Prerna and I have
> >> been working on.  Tracing aids debugging, profiling, and observing
> >> execution via lightweight logging at key points in the code path.

...

> >> 5. Out-of-line trace file write-out
> >>
> >> Owner: Stefan
> >>
> >> Trace buffers are written out to file synchronously.  The vcpu thread
> >> should
> >> not be blocked so an async write-out mechanism is needed.
> >>
> >> 6. Trace file command
> >>
> >> Owner: ?
> >>
> >> Traces are written out to hardcoded /tmp/trace.log.  This must be
> >> configurable.
> >> Tracing at startup time should still be possible so configuration needs to
> >> happen early.

Another feature needed would be a facility to trigger a sync of trace buffer to
file even if the buffer isn't full.

Ananth

Re: [Qemu-devel] Tracing: outstanding tasks

[Stefan Hajnoczi]

On Fri, Jul 2, 2010 at 5:23 AM, Ananth N Mavinakayanahalli
<ananth@in.ibm.com> wrote:
> On Wed, Jun 30, 2010 at 12:51:59PM +0100, Stefan Hajnoczi wrote:
>> On Wed, Jun 30, 2010 at 11:20 AM, Prerna Saxena
>> <prerna@linux.vnet.ibm.com> wrote:
>> > On 06/26/2010 01:36 PM, Stefan Hajnoczi wrote:
>> >>
>> >> Here are the outstanding tasks for QEMU tracing, which Prerna and I have
>> >> been working on.  Tracing aids debugging, profiling, and observing
>> >> execution via lightweight logging at key points in the code path.
>
> ...
>
>> >> 5. Out-of-line trace file write-out
>> >>
>> >> Owner: Stefan
>> >>
>> >> Trace buffers are written out to file synchronously.  The vcpu thread
>> >> should
>> >> not be blocked so an async write-out mechanism is needed.
>> >>
>> >> 6. Trace file command
>> >>
>> >> Owner: ?
>> >>
>> >> Traces are written out to hardcoded /tmp/trace.log.  This must be
>> >> configurable.
>> >> Tracing at startup time should still be possible so configuration needs to
>> >> happen early.
>
> Another feature needed would be a facility to trigger a sync of trace buffer to
> file even if the buffer isn't full.

Good idea.  A related piece that is not in the current tracing branch
is syncing to trace buffer to file on exit.  Right now a partially
filled trace buffer is not written when QEMU exits.

Stefan

Re: [Qemu-devel] Tracing: outstanding tasks

[Stefan Hajnoczi]

I have now moved the roadmap onto the wiki because it becomes hard to
get a sense of the status from an email thread:

http://wiki.qemu.org/Features/Tracing/Roadmap

Stefan