[Qemu-devel] Static tracepoint control via trace-event

[Qemu-devel] Static tracepoint control via trace-event

[Jan Kiszka]

Hi Stefan,

just had a closer look at qemu's new tracing framework. Looks cool,
though it leaves a bit room for improvements. ;)

One quirk I stumbled over quickly was the "disable" tag in trace-events.
It confused me first as qemu starts without any tracepoint enabled by
default and I thought I had to hack the file. Then I read the doc and
wondered which exiting or future backend would come without sufficiently
fast dynamic tracepoint control. Do you have any in mind?

Instead of making it a compile-time switch (except for simpletrace), I
would vote for declaring the simpletrace usage as the only one: disable
sets the default state of the dynamic tracepoint. That way we could use
trace-events to define a useful set of standard, moderate-impact
tracepoints that shall be on. Others will still be available once a
backend is configured, but remain off until enabled during runtime.
Anything else looks like overkill to me.

There are a few more things I have in mind (ftrace backend, enhanced
"-trace" switch, wildcards for enabling tracepoints, and more
tracepoints). Will hopefully come up with patches to address them, but
this may take a while.

Jan

PS: Do you maintain a tracing git tree?

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

[Qemu-devel] Re: Static tracepoint control via trace-event

[Stefan Hajnoczi]

On Tue, Oct 19, 2010 at 03:08:08PM +0200, Jan Kiszka wrote:
> One quirk I stumbled over quickly was the "disable" tag in trace-events.
> It confused me first as qemu starts without any tracepoint enabled by
> default and I thought I had to hack the file. Then I read the doc and
> wondered which exiting or future backend would come without sufficiently
> fast dynamic tracepoint control. Do you have any in mind?
> 
> Instead of making it a compile-time switch (except for simpletrace), I
> would vote for declaring the simpletrace usage as the only one: disable
> sets the default state of the dynamic tracepoint. That way we could use
> trace-events to define a useful set of standard, moderate-impact
> tracepoints that shall be on. Others will still be available once a
> backend is configured, but remain off until enabled during runtime.
> Anything else looks like overkill to me.

The motivation for "disable" producing a nop trace event is that it
allows QEMU builds without certain trace events.  A trace event cannot
simply be removed by deleting its trace-events declaration since there
are calls to its trace_*() function in the source tree.  So this
provided a way to disable trace events before simpletrace supported
enabling/disabling trace events at runtime :).

Today that's no longer an issue for simpletrace and other tracing
backends like LTTng UST and SystemTAP handle disabled trace events well.

I agree that keeping just one meaning for the "disable" keyword is
better.  Perhaps we should keep a separate "nop" keyword to build out
specific trace events.

When would "nop" be handy?  I think an ftrace backend is a good example.
Since an ftrace marker cannot be enabled/disabled at runtime, the only
way to silence unwanted trace events is to "nop" them at compile-time.

> There are a few more things I have in mind (ftrace backend, enhanced
> "-trace" switch, wildcards for enabling tracepoints, and more
> tracepoints). Will hopefully come up with patches to address them, but
> this may take a while.

Sounds great.

> PS: Do you maintain a tracing git tree?

No, I'm reviewing patches as they are posted for qemu-devel.  If the
backlog between mailing list discussion and merge reaches the point
where your patches are suffering conflicts please let me know and I can
maintain one.

For the initial QEMU tracing effort I kept a tree but I stopped after
the patches were accepted into mainline.  The patches I write go
straight to qemu-devel now.

Stefan

[Qemu-devel] Re: Static tracepoint control via trace-event

[Jan Kiszka]

Am 19.10.2010 15:30, Stefan Hajnoczi wrote:
> On Tue, Oct 19, 2010 at 03:08:08PM +0200, Jan Kiszka wrote:
>> One quirk I stumbled over quickly was the "disable" tag in trace-events.
>> It confused me first as qemu starts without any tracepoint enabled by
>> default and I thought I had to hack the file. Then I read the doc and
>> wondered which exiting or future backend would come without sufficiently
>> fast dynamic tracepoint control. Do you have any in mind?
>>
>> Instead of making it a compile-time switch (except for simpletrace), I
>> would vote for declaring the simpletrace usage as the only one: disable
>> sets the default state of the dynamic tracepoint. That way we could use
>> trace-events to define a useful set of standard, moderate-impact
>> tracepoints that shall be on. Others will still be available once a
>> backend is configured, but remain off until enabled during runtime.
>> Anything else looks like overkill to me.
> 
> The motivation for "disable" producing a nop trace event is that it
> allows QEMU builds without certain trace events.  A trace event cannot
> simply be removed by deleting its trace-events declaration since there
> are calls to its trace_*() function in the source tree.  So this
> provided a way to disable trace events before simpletrace supported
> enabling/disabling trace events at runtime :).
> 
> Today that's no longer an issue for simpletrace and other tracing
> backends like LTTng UST and SystemTAP handle disabled trace events well.
> 
> I agree that keeping just one meaning for the "disable" keyword is
> better.  Perhaps we should keep a separate "nop" keyword to build out
> specific trace events.
> 
> When would "nop" be handy?  I think an ftrace backend is a good example.
> Since an ftrace marker cannot be enabled/disabled at runtime, the only
> way to silence unwanted trace events is to "nop" them at compile-time.

Another to-do item is to remove the strange dependency of tracing
managements features on CONFIG_SIMPLE_TRACE. That way the monitor
commands and a to-be-added command line option to control individual
tracepoints could of course also be used by an ftrace backend. I bet the
DTrace backend will like to see this as well.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Re: Static tracepoint control via trace-event

[Stefan Hajnoczi]

On Tue, Oct 19, 2010 at 2:46 PM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> Am 19.10.2010 15:30, Stefan Hajnoczi wrote:
>> On Tue, Oct 19, 2010 at 03:08:08PM +0200, Jan Kiszka wrote:
>>> One quirk I stumbled over quickly was the "disable" tag in trace-events.
>>> It confused me first as qemu starts without any tracepoint enabled by
>>> default and I thought I had to hack the file. Then I read the doc and
>>> wondered which exiting or future backend would come without sufficiently
>>> fast dynamic tracepoint control. Do you have any in mind?
>>>
>>> Instead of making it a compile-time switch (except for simpletrace), I
>>> would vote for declaring the simpletrace usage as the only one: disable
>>> sets the default state of the dynamic tracepoint. That way we could use
>>> trace-events to define a useful set of standard, moderate-impact
>>> tracepoints that shall be on. Others will still be available once a
>>> backend is configured, but remain off until enabled during runtime.
>>> Anything else looks like overkill to me.
>>
>> The motivation for "disable" producing a nop trace event is that it
>> allows QEMU builds without certain trace events.  A trace event cannot
>> simply be removed by deleting its trace-events declaration since there
>> are calls to its trace_*() function in the source tree.  So this
>> provided a way to disable trace events before simpletrace supported
>> enabling/disabling trace events at runtime :).
>>
>> Today that's no longer an issue for simpletrace and other tracing
>> backends like LTTng UST and SystemTAP handle disabled trace events well.
>>
>> I agree that keeping just one meaning for the "disable" keyword is
>> better.  Perhaps we should keep a separate "nop" keyword to build out
>> specific trace events.
>>
>> When would "nop" be handy?  I think an ftrace backend is a good example.
>> Since an ftrace marker cannot be enabled/disabled at runtime, the only
>> way to silence unwanted trace events is to "nop" them at compile-time.
>
> Another to-do item is to remove the strange dependency of tracing
> managements features on CONFIG_SIMPLE_TRACE. That way the monitor
> commands and a to-be-added command line option to control individual
> tracepoints could of course also be used by an ftrace backend. I bet the
> DTrace backend will like to see this as well.

If there is a programmatic way of inspecting and toggling trace events
from inside an instrumented process, then yes.  If this is possible
with SystemTAP we should think about it now before QMP tracing
commands become available in a release.

Stefan

Re: [Qemu-devel] Re: Static tracepoint control via trace-event

[Daniel P. Berrange]

On Tue, Oct 19, 2010 at 03:46:35PM +0200, Jan Kiszka wrote:
> Am 19.10.2010 15:30, Stefan Hajnoczi wrote:
> > On Tue, Oct 19, 2010 at 03:08:08PM +0200, Jan Kiszka wrote:
> >> One quirk I stumbled over quickly was the "disable" tag in trace-events.
> >> It confused me first as qemu starts without any tracepoint enabled by
> >> default and I thought I had to hack the file. Then I read the doc and
> >> wondered which exiting or future backend would come without sufficiently
> >> fast dynamic tracepoint control. Do you have any in mind?
> >>
> >> Instead of making it a compile-time switch (except for simpletrace), I
> >> would vote for declaring the simpletrace usage as the only one: disable
> >> sets the default state of the dynamic tracepoint. That way we could use
> >> trace-events to define a useful set of standard, moderate-impact
> >> tracepoints that shall be on. Others will still be available once a
> >> backend is configured, but remain off until enabled during runtime.
> >> Anything else looks like overkill to me.
> > 
> > The motivation for "disable" producing a nop trace event is that it
> > allows QEMU builds without certain trace events.  A trace event cannot
> > simply be removed by deleting its trace-events declaration since there
> > are calls to its trace_*() function in the source tree.  So this
> > provided a way to disable trace events before simpletrace supported
> > enabling/disabling trace events at runtime :).
> > 
> > Today that's no longer an issue for simpletrace and other tracing
> > backends like LTTng UST and SystemTAP handle disabled trace events well.
> > 
> > I agree that keeping just one meaning for the "disable" keyword is
> > better.  Perhaps we should keep a separate "nop" keyword to build out
> > specific trace events.
> > 
> > When would "nop" be handy?  I think an ftrace backend is a good example.
> > Since an ftrace marker cannot be enabled/disabled at runtime, the only
> > way to silence unwanted trace events is to "nop" them at compile-time.
> 
> Another to-do item is to remove the strange dependency of tracing
> managements features on CONFIG_SIMPLE_TRACE. That way the monitor
> commands and a to-be-added command line option to control individual
> tracepoints could of course also be used by an ftrace backend. I bet the
> DTrace backend will like to see this as well.

I don't see a need for any monitor commands or command line options
for the DTrace backend, since everything is completely dynamically
controlled based on the tracing scripts the user is running. 

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: Static tracepoint control via trace-event

[Jan Kiszka]

Am 19.10.2010 16:12, Daniel P. Berrange wrote:
> On Tue, Oct 19, 2010 at 03:46:35PM +0200, Jan Kiszka wrote:
>> Am 19.10.2010 15:30, Stefan Hajnoczi wrote:
>>> On Tue, Oct 19, 2010 at 03:08:08PM +0200, Jan Kiszka wrote:
>>>> One quirk I stumbled over quickly was the "disable" tag in trace-events.
>>>> It confused me first as qemu starts without any tracepoint enabled by
>>>> default and I thought I had to hack the file. Then I read the doc and
>>>> wondered which exiting or future backend would come without sufficiently
>>>> fast dynamic tracepoint control. Do you have any in mind?
>>>>
>>>> Instead of making it a compile-time switch (except for simpletrace), I
>>>> would vote for declaring the simpletrace usage as the only one: disable
>>>> sets the default state of the dynamic tracepoint. That way we could use
>>>> trace-events to define a useful set of standard, moderate-impact
>>>> tracepoints that shall be on. Others will still be available once a
>>>> backend is configured, but remain off until enabled during runtime.
>>>> Anything else looks like overkill to me.
>>>
>>> The motivation for "disable" producing a nop trace event is that it
>>> allows QEMU builds without certain trace events.  A trace event cannot
>>> simply be removed by deleting its trace-events declaration since there
>>> are calls to its trace_*() function in the source tree.  So this
>>> provided a way to disable trace events before simpletrace supported
>>> enabling/disabling trace events at runtime :).
>>>
>>> Today that's no longer an issue for simpletrace and other tracing
>>> backends like LTTng UST and SystemTAP handle disabled trace events well.
>>>
>>> I agree that keeping just one meaning for the "disable" keyword is
>>> better.  Perhaps we should keep a separate "nop" keyword to build out
>>> specific trace events.
>>>
>>> When would "nop" be handy?  I think an ftrace backend is a good example.
>>> Since an ftrace marker cannot be enabled/disabled at runtime, the only
>>> way to silence unwanted trace events is to "nop" them at compile-time.
>>
>> Another to-do item is to remove the strange dependency of tracing
>> managements features on CONFIG_SIMPLE_TRACE. That way the monitor
>> commands and a to-be-added command line option to control individual
>> tracepoints could of course also be used by an ftrace backend. I bet the
>> DTrace backend will like to see this as well.
> 
> I don't see a need for any monitor commands or command line options
> for the DTrace backend, since everything is completely dynamically
> controlled based on the tracing scripts the user is running. 

Ah, it's all dynamic probing, you just need the marks. OK, was a bad
example. :)

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Static tracepoint control via trace-event

[Daniel P. Berrange]

On Tue, Oct 19, 2010 at 03:08:08PM +0200, Jan Kiszka wrote:
> Hi Stefan,
> 
> just had a closer look at qemu's new tracing framework. Looks cool,
> though it leaves a bit room for improvements. ;)
> 
> One quirk I stumbled over quickly was the "disable" tag in trace-events.
> It confused me first as qemu starts without any tracepoint enabled by
> default and I thought I had to hack the file. Then I read the doc and
> wondered which exiting or future backend would come without sufficiently
> fast dynamic tracepoint control. Do you have any in mind?
> 
> Instead of making it a compile-time switch (except for simpletrace), I
> would vote for declaring the simpletrace usage as the only one: disable
> sets the default state of the dynamic tracepoint. That way we could use
> trace-events to define a useful set of standard, moderate-impact
> tracepoints that shall be on. Others will still be available once a
> backend is configured, but remain off until enabled during runtime.
> Anything else looks like overkill to me.

FYI with the DTrace/SystemTAP backend I posted yesterday, the 'disable'
keyword is effectively completely ignored. All tracepoints are disabled
when QEMU is running normally. Only when a end user runs a dtrace script
that references a QEMU tracepoint, is that specific tracepoint enabled.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Static tracepoint control via trace-event

[Stefan Hajnoczi]

On Tue, Oct 19, 2010 at 2:36 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Tue, Oct 19, 2010 at 03:08:08PM +0200, Jan Kiszka wrote:
>> Hi Stefan,
>>
>> just had a closer look at qemu's new tracing framework. Looks cool,
>> though it leaves a bit room for improvements. ;)
>>
>> One quirk I stumbled over quickly was the "disable" tag in trace-events.
>> It confused me first as qemu starts without any tracepoint enabled by
>> default and I thought I had to hack the file. Then I read the doc and
>> wondered which exiting or future backend would come without sufficiently
>> fast dynamic tracepoint control. Do you have any in mind?
>>
>> Instead of making it a compile-time switch (except for simpletrace), I
>> would vote for declaring the simpletrace usage as the only one: disable
>> sets the default state of the dynamic tracepoint. That way we could use
>> trace-events to define a useful set of standard, moderate-impact
>> tracepoints that shall be on. Others will still be available once a
>> backend is configured, but remain off until enabled during runtime.
>> Anything else looks like overkill to me.
>
> FYI with the DTrace/SystemTAP backend I posted yesterday, the 'disable'
> keyword is effectively completely ignored. All tracepoints are disabled
> when QEMU is running normally. Only when a end user runs a dtrace script
> that references a QEMU tracepoint, is that specific tracepoint enabled.

I think that makes sense for external trace backends.  DTrace can
launch a process for you with the probes you want enabled from the
start.  The simpletrace backend can't really do this so probes can be
enabled/disabled at compile-time (e.g. early startup tracing).

Stefan

Re: [Qemu-devel] Static tracepoint control via trace-event

[Jan Kiszka]

Am 19.10.2010 15:52, Stefan Hajnoczi wrote:
> On Tue, Oct 19, 2010 at 2:36 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
>> On Tue, Oct 19, 2010 at 03:08:08PM +0200, Jan Kiszka wrote:
>>> Hi Stefan,
>>>
>>> just had a closer look at qemu's new tracing framework. Looks cool,
>>> though it leaves a bit room for improvements. ;)
>>>
>>> One quirk I stumbled over quickly was the "disable" tag in trace-events.
>>> It confused me first as qemu starts without any tracepoint enabled by
>>> default and I thought I had to hack the file. Then I read the doc and
>>> wondered which exiting or future backend would come without sufficiently
>>> fast dynamic tracepoint control. Do you have any in mind?
>>>
>>> Instead of making it a compile-time switch (except for simpletrace), I
>>> would vote for declaring the simpletrace usage as the only one: disable
>>> sets the default state of the dynamic tracepoint. That way we could use
>>> trace-events to define a useful set of standard, moderate-impact
>>> tracepoints that shall be on. Others will still be available once a
>>> backend is configured, but remain off until enabled during runtime.
>>> Anything else looks like overkill to me.
>>
>> FYI with the DTrace/SystemTAP backend I posted yesterday, the 'disable'
>> keyword is effectively completely ignored. All tracepoints are disabled
>> when QEMU is running normally. Only when a end user runs a dtrace script
>> that references a QEMU tracepoint, is that specific tracepoint enabled.
> 
> I think that makes sense for external trace backends.  DTrace can
> launch a process for you with the probes you want enabled from the
> start.  The simpletrace backend can't really do this so probes can be
> enabled/disabled at compile-time (e.g. early startup tracing).

Once we have "-trace events=...", defining the list of active
tracepoints before starting qemu will be trivial (e.g. via a config
file). Of course, this requires that all tracepoints are built-in...

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Tracing block devices (was: Re: [Qemu-devel] Static tracepoint control via trace-event)

[Richard W.M. Jones]

On Tue, Oct 19, 2010 at 03:59:36PM +0200, Jan Kiszka wrote:
> Once we have "-trace events=...", defining the list of active
> tracepoints before starting qemu will be trivial (e.g. via a config
> file). Of course, this requires that all tracepoints are built-in...

Sorry that I've not been following this very closely, but does this
sort of thing allow tracing reads and writes to block devices?  Am I
right in thinking that if a tracepoint existed in the right place, one
could get a log file from that which could be post-processed in
another tool?

cf:
http://rwmj.wordpress.com/2010/10/05/visualizing-reads-writes-and-alignment/#content

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

[Qemu-devel] Re: Tracing block devices

[Jan Kiszka]

Am 19.10.2010 16:29, Richard W.M. Jones wrote:
> On Tue, Oct 19, 2010 at 03:59:36PM +0200, Jan Kiszka wrote:
>> Once we have "-trace events=...", defining the list of active
>> tracepoints before starting qemu will be trivial (e.g. via a config
>> file). Of course, this requires that all tracepoints are built-in...
> 
> Sorry that I've not been following this very closely, but does this
> sort of thing allow tracing reads and writes to block devices?  Am I
> right in thinking that if a tracepoint existed in the right place, one
> could get a log file from that which could be post-processed in
> another tool?
> 
> cf:
> http://rwmj.wordpress.com/2010/10/05/visualizing-reads-writes-and-alignment/#content
> 
> Rich.
> 

Yes. The block layer is instrumented, not sure if already sufficiently,
but you may simply want to try the simpletrace backend and inspect the
result via its postprocessor (simpletrace.py).

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: Tracing block devices (was: Re: [Qemu-devel] Static tracepoint control via trace-event)

[Stefan Hajnoczi]

On Tue, Oct 19, 2010 at 03:29:51PM +0100, Richard W.M. Jones wrote:
> On Tue, Oct 19, 2010 at 03:59:36PM +0200, Jan Kiszka wrote:
> > Once we have "-trace events=...", defining the list of active
> > tracepoints before starting qemu will be trivial (e.g. via a config
> > file). Of course, this requires that all tracepoints are built-in...
> 
> Sorry that I've not been following this very closely, but does this
> sort of thing allow tracing reads and writes to block devices?  Am I
> right in thinking that if a tracepoint existed in the right place, one
> could get a log file from that which could be post-processed in
> another tool?
> 
> cf:
> http://rwmj.wordpress.com/2010/10/05/visualizing-reads-writes-and-alignment/#content

Definitely, here is the commit that added bdrv_aio_writev/bdrv_aio_readv
tracing.  bdrv_aio_multiwrite has been traced for a while.

http://patchwork.ozlabs.org/patch/66843/

As an example, I use the following script to find all write requests
that touch a given region.  This is very useful for debugging image
corruptions given a trace file:

The usage is:

find_overlapping_io.py <bs> <sector_num> <nb_sectors>

where bs is the block driver state pointer, sector_num is the starting
sector address, and nb_sectors is the number of sectors.

#!/usr/bin/env python
import sys

def trace_filter(fobj, event, keys):
    for line in fobj:
        fields = line.strip().split()
        if fields[0] != event:
            continue

        attrs = dict([(k, v) for k, v in (x.split('=') for x in fields[2:])])
        match = True
        for k, v in keys.iteritems():
            if k not in attrs:
                match = False
                break
            if attrs[k] != v:
                match = False
                break

        if match:
            yield attrs

def intersection(a_sector_num, a_nb_sectors, b_sector_num, b_nb_sectors):
    return not (a_sector_num + a_nb_sectors <= b_sector_num or \
                b_sector_num + b_nb_sectors <= a_sector_num)

bs, sector_num, nb_sectors = sys.argv[1:]
sector_num = int(sector_num, 0)
nb_sectors = int(nb_sectors, 0)

for req in trace_filter(sys.stdin, 'bdrv_aio_writev', {'bs': bs}):
    if intersection(sector_num, nb_sectors, int(req['sector_num'], 0), int(req['nb_sectors'], 0)):
        print req

Stefan

Re: Tracing block devices (was: Re: [Qemu-devel] Static tracepoint control via trace-event)

[Christoph Hellwig]

On Tue, Oct 19, 2010 at 03:29:51PM +0100, Richard W.M. Jones wrote:
> On Tue, Oct 19, 2010 at 03:59:36PM +0200, Jan Kiszka wrote:
> > Once we have "-trace events=...", defining the list of active
> > tracepoints before starting qemu will be trivial (e.g. via a config
> > file). Of course, this requires that all tracepoints are built-in...
> 
> Sorry that I've not been following this very closely, but does this
> sort of thing allow tracing reads and writes to block devices?  Am I
> right in thinking that if a tracepoint existed in the right place, one
> could get a log file from that which could be post-processed in
> another tool?

I don't think it does.  But I have a patch that I need to update which
outputs traces in the form of the Linux blktrace block I/O tracing tool,
which does exactly what you need (and more).  You can simply use
all the blktrace-enabled tools, including seekwatcher on it.

Re: Tracing block devices (was: Re: [Qemu-devel] Static tracepoint control via trace-event)

[Richard W.M. Jones]

On Thu, Oct 21, 2010 at 07:20:27AM +0200, Christoph Hellwig wrote:
> On Tue, Oct 19, 2010 at 03:29:51PM +0100, Richard W.M. Jones wrote:
> > On Tue, Oct 19, 2010 at 03:59:36PM +0200, Jan Kiszka wrote:
> > > Once we have "-trace events=...", defining the list of active
> > > tracepoints before starting qemu will be trivial (e.g. via a config
> > > file). Of course, this requires that all tracepoints are built-in...
> > 
> > Sorry that I've not been following this very closely, but does this
> > sort of thing allow tracing reads and writes to block devices?  Am I
> > right in thinking that if a tracepoint existed in the right place, one
> > could get a log file from that which could be post-processed in
> > another tool?
> 
> I don't think it does.  But I have a patch that I need to update which
> outputs traces in the form of the Linux blktrace block I/O tracing tool,
> which does exactly what you need (and more).  You can simply use
> all the blktrace-enabled tools, including seekwatcher on it.

It's on the to-do list to use blktrace instead, so a patch
which changed qemu to produce blktrace-like output would
be useful.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/

Re: Tracing block devices (was: Re: [Qemu-devel] Static tracepoint control via trace-event)

[Stefan Hajnoczi]

On Thu, Oct 21, 2010 at 6:20 AM, Christoph Hellwig <hch@lst.de> wrote:
> On Tue, Oct 19, 2010 at 03:29:51PM +0100, Richard W.M. Jones wrote:
>> On Tue, Oct 19, 2010 at 03:59:36PM +0200, Jan Kiszka wrote:
>> > Once we have "-trace events=...", defining the list of active
>> > tracepoints before starting qemu will be trivial (e.g. via a config
>> > file). Of course, this requires that all tracepoints are built-in...
>>
>> Sorry that I've not been following this very closely, but does this
>> sort of thing allow tracing reads and writes to block devices?  Am I
>> right in thinking that if a tracepoint existed in the right place, one
>> could get a log file from that which could be post-processed in
>> another tool?
>
> I don't think it does.  But I have a patch that I need to update which
> outputs traces in the form of the Linux blktrace block I/O tracing tool,
> which does exactly what you need (and more).  You can simply use
> all the blktrace-enabled tools, including seekwatcher on it.

There are trace events in qemu.git today that let you trace block
read/write operations (device, sector, length, r/w information is
there).  Look at bdrv_aio_readv and bdrv_aio_writev in the
trace-events file.

I can see the value of using existing tools to parse blktrace data.
On the other hand blktrace is taylored to Linux's block layer and QEMU
can only map to a subset of it.  I'm interested in your patch too, is
it a new block protocol or does it hook directly into block.c?

Stefan

Re: Tracing block devices (was: Re: [Qemu-devel] Static tracepoint control via trace-event)

[Daniel P. Berrange]

On Tue, Oct 19, 2010 at 03:29:51PM +0100, Richard W.M. Jones wrote:
> On Tue, Oct 19, 2010 at 03:59:36PM +0200, Jan Kiszka wrote:
> > Once we have "-trace events=...", defining the list of active
> > tracepoints before starting qemu will be trivial (e.g. via a config
> > file). Of course, this requires that all tracepoints are built-in...
> 
> Sorry that I've not been following this very closely, but does this
> sort of thing allow tracing reads and writes to block devices?  Am I
> right in thinking that if a tracepoint existed in the right place, one
> could get a log file from that which could be post-processed in
> another tool?
> 
> cf:
> http://rwmj.wordpress.com/2010/10/05/visualizing-reads-writes-and-alignment/#content

While having a static tracepoint in the right place would be best, it
is not strictly neccessary with a tool like DTrace/SystemTAP. With the
qemu debuginfo available, those tools can dynamically insert a probe
into any QEMU function at any point in the code. So you could easily
replace your QEMU patch from that blog post with a simple trace script
and get the same info dynamically. The benefit of static markers is 
that they can provide standard named probe point + args,  which are 
stable long term, even as the code is re-factored/renamed/moved, etc.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|