Re: Benchmarks of kernel tracing options (ftrace, ktrace, lttng and perf)

[Mathieu Desnoyers <mathieu.desnoyers@efficios.com>]

* Michael Rubin (mrubin@google.com) wrote:
> On Thu, Oct 28, 2010 at 12:27 PM, Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:
> > * David Sharp (dhsharp@google.com) wrote:
> > Hrm, not quite fair actually. LTTng enables a thread flag that causes
> > syscall_trace to be called on sycall entry/exit by default. Please try with the
> > following patch to remove the extra syscall overhead.
> 
> We started this thread is an attempt to get the ball rolling on
> keeping track of these metrics. They seem important for both Google
> and Linux as a whole. We published all our work and benchmarks with
> hopes that the owners could comment and also run the tests themselves.
> 
> Matthieu it might make more sense for you to find the" fair" setup and
> publish that, than take the extra time to go back and forth with
> David. Once you have the config that best represents what users will
> see and you have numbers to go with it, we can then run it in our
> environment. I worry you will be forced to send us patch and patch and
> patch and we won't have the resources to do a good job in responding
> to them. Which is not fair to you.

Nah, it's really a matter of fine-tuning really. I already performed these
benchmarks, have published the results, and know what to expect, so I figured
out quickly that something was wrong with your setup. Disabling the extra
syscall tracing is the only detail I see that is missing for fair comparison of
LTTng with your setup.

For the "fair" setup to compare ring buffers, I would personally tend to say
that going through a system call might not be the way I would benchmark the ring
buffer per se, but I must admit that the benchmark you are doing is very useful
for evaluation of system call entry/exit tracing. On this particular aspect, I
have always admitted that lttng is not super-fast, but it actually takes the
same overhead as the Linux kernel syscall tracing has today. But given it is
always enabled when lttng tracing runs, you happened to add this overhead to the
lttng ring buffer inappropriately.

I'm willing to help David with the setup, and I'm sure it won't take long. By
the way, I'm currently porting lttng to the new Generic Ring Buffer Library,
which might be even slightly faster than the current lttng code (and faster than
ftrace with the ring buffer benchmark Steven wrote). So when I'm done porting
lttng to it, it might be worthwhile to benchmark the Generic Ring Buffer too.

Thank you,

Mathieu

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com