Re: [ltt-dev] LTTng 0.27, vmap-less buffering and splice()

[Mathieu Desnoyers <compudj@krystal.dyndns.org>]

I did some performance testing between LTTng 0.26 (uses vmap buffers)
and LTTng 0.31 (vmap-less buffers). I think performance got degraded in
the process. Those tests only write trace data in circular buffers
("overwrite mode").

No tracing, kernel 2.6.27-rc8 :
tbench : 1862.24

LTTng 0.26, kernel 2.6.27-rc7, flight recorder, default size buffers :
tbench : 1156.37 MB/s

LTTng 0.31, kernel 2.6.27-rc8, flight recorder, default size buffers :
tbench : 942.72 MB/s

For those of the LTT community interested in LTTng performance, I think
it's worth having a look at the ltt-relay-alloc.patch implementation to
see if there is any obvious performance degradation source. I'll play a
bit with the implementation to see where it comes from.

Mathieu

* Mathieu Desnoyers (mathieu.desnoyers@polymtl.ca) wrote:
> Hi,
> 
> I have reworked the underlying buffering mechanism LTTng uses in the
> past week. I took "relay", changed its vmap() buffers for my own linked
> list of buffers (not vmaped), made read/write wrappers which support
> writing data larger than a page size and writing across page boundaries,
> and finally managed to create a splice_read file operation which
> supports that. I finally changed lttd to make it use splice() instead of
> a mmap() and... it worked! :-) (after a bit a debugging, clearly)
> 
> This is all in LTTng 0.27 and ltt-control 0.53 (for the lttd part).
> As this is an important change, testing is very welcome. If you are
> interested in looking in the inner details of the buffering mechanism I
> just did, you might also want to enable the "check for random buffer
> access" option in the menuconfig. It will generate warnings when offset
> accesses more than a page away from the previous done is requested by
> the client. Cases such as large data write and reentrancy over the
> tracer code will generate a few "cache misses", but it's supposed to be
> rare, and therefore not a performance concern. This self-checking
> feature has proven to be very useful in the early development stages,
> and I think it's also useful if any other tracer client want to use it :
> it helps finding lack of reference locality a tracer client might have.
> 
> I am also very interested in getting numbers comparing the performance
> of the new buffering infrastructure with the previous one. Having much
> less TLB impact should improve performance.
> 
> It's all available in the git tree :
> http://git.kernel.org/?p=linux/kernel/git/compudj/linux-2.6-lttng.git;a=summary
> 
> And the userspace packages available at http://ltt.polymtl.ca (see the
> "QUICKSTART")
> 
> Cheers,
> 
> Mathieu
> 
> -- 
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> 
> _______________________________________________
> ltt-dev mailing list
> ltt-dev@lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68