Re: [PATCH 0/6] blkin (LTTng + Zipkin) tracing

[Andrew Shewmaker <agshew@gmail.com>]

On Thu, Nov 13, 2014 at 08:14:48AM -0800, Sage Weil wrote:
> On Wed, 12 Nov 2014, Andrew Shewmaker wrote:
<snip>
> > In general, Blkin implements the tracing semantics described in the Dapper
> > paper http://static.googleusercontent.com/media/research.google.com/el/pubs/archive/36356.pdf
> > in order to trace the causal relationships between the different
> > processing phases that an IO request may trigger. The goal is an end-to-end
> > visualisation of the request's route in the system, accompanied by information
> > concerning latencies in each processing phase. Thanks to LTTng this can happen
> > with a minimal overhead and in realtime. In order to visualize the results Blkin 
> > was integrated with Twitter's Zipkin http://twitter.github.io/zipkin/
> > (which is a tracing system entirely based on Dapper).
> > 
> > These patches can also be found in https://github.com/agshew/ceph/tree/wip-blkin
> 
> This looks great!  Do you mind opening a github pull request from that 
> branch?  It's a bit more convenient for capturing review.

I'll do that, but first I need to make changes to autoconf/automake
for blkin. It isn't actually building.

I had been going down the road of treating it simply 
as a separate package from ceph, then decided to include it as a 
submodule. My branch only built on my system because I had already
installed blkin separately.

<snip>
> > In the immediate future I plan to:
> > 
> >  - push a wip-blkin branch to github.com/ceph and take advantage of gitbuilder test/qa
> >  - move the changes forward to ceph:master
> >  - add Andreas' tracepoints https://github.com/ceph/ceph/pull/2877 using Blkin
> >    and investigate how easy it is to select the level of tracing detail
> > 
> > Questions:
> > 
> > 1. Did I split the patches into sensible groups?
> 
> 1 could be broken into the build changes and the msg/optracker code.  It 
> looks like it unconditionally links against zipkin-cpp now, which we 
> probably don't want.  Unless blkin is statically linked or something, but 
> I don't see anything in the patch that would do that yet.  In any case, 
> having the build stuff in a separate patch helps.

Right. Makes sense. I'll split that out for version 2.

 
> The split for the rest looks fine.  Need to look at the changes to osd 
> init carefully as it is a bit delicate.
> 
> > 2. How low is LTTng's overhead? Is it entirely eliminated when not enabled?
> > 
> > Do we need to take advantage of something like the Linux kernel's CONFIG_DYNAMIC_FTRACE
> > trick, where a special mcount() function is converted back and forth between 
> > a NOP and trace calls? See http://lwn.net/Articles/365835/ for a little more 
> > detail.
> 
> I always assumed that lttng was doing something like this, but I don't see 
> a clear explanation of what an inactive tracepoint looks like anywhere..

Yeah, I suppose they must, but I couldn't find a nice short
explanation either. LWN has a couple of articles that don't go deeply
into it. At most, they mention the use of asm gotos
(http://lwn.net/Articles/491543/).