Re: Ιnstrumenting RADOS with Zipkin + LTTng

Re: Ιnstrumenting RADOS with Zipkin + LTTng

[Andrew Shewmaker]

Hi everyone,

I'm Andrew, a new Ceph intern, and I'll be working to get Marios'
Zipkin + LTTng repo into a merge-able state, as Sam asked about
(http://www.spinics.net/lists/ceph-devel/msg20024.html).

I've exchanged email with Marios, and he is interested in 
helping me with it in his spare time. One of the important 
things he mentioned is that his blkin library only works 
with LTTng 2.4. Other versions experience deadlocks, and 
he'll work to resolve them in 2.5.

I'm still working on getting a ceph development environment 
set up, so I haven't tested the tracepoints Marios added.
I have built an RPM spec file for blkin, tested it against 
lttng-ust 2.3 and 2.4, and built (but not tested) Marios'
ceph branch.

Should I put effort into adding a "--with-blkin" or "--with-zipkin"
option to autoconf?

I've made a first attempt at dividing the changes into
logically grouped patches: common blkin infrastructure, 
osd, and rados. Does that sound reasonable? Or should 
I not bother separating osd and rados changes?

I saw a couple instances of extraneous whitespace/newlines
I'll clean up. What other issues should I look for?

After they're cleaned up, would it be best for me to submit these 
patches to the list, or just point to a github repo?

Thanks,

Andrew Shewmaker

Re: Ιnstrumenting RADOS with Zipkin + LTTng

[Sage Weil]

Hi Andrew!

On Thu, 30 Oct 2014, Andrew Shewmaker wrote:
> Hi everyone,
> 
> I'm Andrew, a new Ceph intern, and I'll be working to get Marios'
> Zipkin + LTTng repo into a merge-able state, as Sam asked about
> (http://www.spinics.net/lists/ceph-devel/msg20024.html).

Awesome!

> I've exchanged email with Marios, and he is interested in 
> helping me with it in his spare time. One of the important 
> things he mentioned is that his blkin library only works 
> with LTTng 2.4. Other versions experience deadlocks, and 
> he'll work to resolve them in 2.5.
> 
> I'm still working on getting a ceph development environment 
> set up, so I haven't tested the tracepoints Marios added.
> I have built an RPM spec file for blkin, tested it against 
> lttng-ust 2.3 and 2.4, and built (but not tested) Marios'
> ceph branch.
> 
> Should I put effort into adding a "--with-blkin" or "--with-zipkin"
> option to autoconf?

Is it statically or dynamically linked?

Probably a first step is to get it in the gitbuilder (test/qa) builds.  

> I've made a first attempt at dividing the changes into
> logically grouped patches: common blkin infrastructure, 
> osd, and rados. Does that sound reasonable? Or should 
> I not bother separating osd and rados changes?

Separate patches is good for review etc.

> I saw a couple instances of extraneous whitespace/newlines
> I'll clean up. What other issues should I look for?
> 
> After they're cleaned up, would it be best for me to submit these 
> patches to the list, or just point to a github repo?

A github pull request is best as that captures the review nicely and is 
how most of the code is going into ceph.git.

Thanks!
sage

Re: Ιnstrumenting RADOS with Zipkin + LTTng

[Andrew Shewmaker]

On Fri, Nov 07, 2014 at 02:32:51AM -0800, Sage Weil wrote:
> 
> On Thu, 30 Oct 2014, Andrew Shewmaker wrote:
> > 
> > Should I put effort into adding a "--with-blkin" or "--with-zipkin"
> > option to autoconf?
> 
> Is it statically or dynamically linked?
> 
> Probably a first step is to get it in the gitbuilder (test/qa) builds.  

Okay, I'll work on getting it in gitbuilder.

It's dynamically linked, and I imagine blkin will be useful outside 
of Ceph, at least for apps built on top of Ceph.

So before getting it in gitbuilder, I'm thinking I'll add blkin as a
submodule that is always built. Then after it's in gitbuilder I'll put
in autoconf support and build libblkin and libblkin-dev packages?
Would you prefer autoconf to enable blkin by default?

Andrew

Re: Ιnstrumenting RADOS with Zipkin + LTTng

[Mark Nelson]

On 10/30/2014 01:49 PM, Andrew Shewmaker wrote:
> Hi everyone,
>
> I'm Andrew, a new Ceph intern, and I'll be working to get Marios'
> Zipkin + LTTng repo into a merge-able state, as Sam asked about
> (http://www.spinics.net/lists/ceph-devel/msg20024.html).
>
> I've exchanged email with Marios, and he is interested in
> helping me with it in his spare time. One of the important
> things he mentioned is that his blkin library only works
> with LTTng 2.4. Other versions experience deadlocks, and
> he'll work to resolve them in 2.5.
>
> I'm still working on getting a ceph development environment
> set up, so I haven't tested the tracepoints Marios added.
> I have built an RPM spec file for blkin, tested it against
> lttng-ust 2.3 and 2.4, and built (but not tested) Marios'
> ceph branch.
>
> Should I put effort into adding a "--with-blkin" or "--with-zipkin"
> option to autoconf?
>
> I've made a first attempt at dividing the changes into
> logically grouped patches: common blkin infrastructure,
> osd, and rados. Does that sound reasonable? Or should
> I not bother separating osd and rados changes?
>
> I saw a couple instances of extraneous whitespace/newlines
> I'll clean up. What other issues should I look for?
>
> After they're cleaned up, would it be best for me to submit these
> patches to the list, or just point to a github repo?
>
> Thanks,
>
> Andrew Shewmaker

Just wanted to say I'm really looking forward to this. :)

Mark

> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Ιnstrumenting RADOS with Zipkin + LTTng

[Matt W. Benjamin]

Hi,

Just wanted to say, Ali here has been experimenting with these as well, and, although
I don't have any specifics yet, we'll contribute anything we come up with that looks
helpful.

Thanks all very much for the work,

Matt

----- "Mark Nelson" <mark.nelson@inktank.com> wrote:

> On 10/30/2014 01:49 PM, Andrew Shewmaker wrote:
> > Hi everyone,
> >
> > I'm Andrew, a new Ceph intern, and I'll be working to get Marios'
> > Zipkin + LTTng repo into a merge-able state, as Sam asked about
> > (http://www.spinics.net/lists/ceph-devel/msg20024.html).

-- 
Matt Benjamin
CohortFS, LLC.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://cohortfs.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309 

Ιnstrumenting RADOS with Zipkin + LTTng

[Marios-Evaggelos Kogias]

Hello all,

my name is Marios Kogias and I am a student at the National Technical
University of Athens. As part of my diploma thesis and my participation in
Google Summer of Code 2014 (in the LTTng organization) I am working on a
low-overhead tracing infrastructure for distributed systems. I am also
collaborating with the Synnefo team (https://www.synnefo.org/) and especially
with Vangelis Koukis, Constantinos Venetsanopoulos and Filippos Giannakos (cc)

Some time ago, we started experimenting with RADOS instrumentation
using LTTng and
we noticed that there are similar endeavours in the Ceph github repository [1].

However, unlike your approach, we are following an annotation-based tracing
schema, which enables us to track a specific request from the time it enters
the system at higher levels till it is finally served by RADOS.

In general, we try to implement the tracing semantics described in the Dapper
paper [2] in order to trace the causal relationships between the different
processing phases that an IO request may trigger. Our target is an end-to-end
visualisation of the request's route in the system, accompanied by information
concerning latencies in each processing phase. Thanks to LTTng this can happen
with a minimal overhead and in realtime. In order to visualize the results we
have integrated Twitter's Zipkin [3], (which is a tracing system
entirely based on
Dapper) with LTTng.

You can find a proof of concept of what we've done so far here:

http://snf-551656.vm.okeanos.grnet.gr:8080/traces/0b554b8a48cb3e84?serviceName=MOSDOp

In the above link you can see the trace of a write request served by a RADOS
pool with replication level set to 3 (two replicas).

We'd love to have early feedback and comments from you guys too,
so that we can incorporate useful recommendations. You can find all
the relevant code
here[5][6]. If you have any questions or you wish to experiment with the
project please do not hesitate to contact us.

Kind regards,
Marios

[1]https://github.com/ceph/ceph/tree/wip-lttng
[2]http://static.googleusercontent.com/media/research.google.com/el//pubs/archive/36356.pdf
[3]http://twitter.github.io/zipkin/
[4] https://github.com/marioskogias/blkin
[5] https://github.com/marioskogias/babeltrace-plugins

Re: Ιnstrumenting RADOS with Zipkin + LTTng

[Samuel Just]

Reading!
-Sam

On Fri, Aug 1, 2014 at 9:28 AM, Marios-Evaggelos Kogias
<marioskogias@gmail.com> wrote:
> Hello all,
>
> my name is Marios Kogias and I am a student at the National Technical
> University of Athens. As part of my diploma thesis and my participation in
> Google Summer of Code 2014 (in the LTTng organization) I am working on a
> low-overhead tracing infrastructure for distributed systems. I am also
> collaborating with the Synnefo team (https://www.synnefo.org/) and especially
> with Vangelis Koukis, Constantinos Venetsanopoulos and Filippos Giannakos (cc)
>
> Some time ago, we started experimenting with RADOS instrumentation
> using LTTng and
> we noticed that there are similar endeavours in the Ceph github repository [1].
>
> However, unlike your approach, we are following an annotation-based tracing
> schema, which enables us to track a specific request from the time it enters
> the system at higher levels till it is finally served by RADOS.
>
> In general, we try to implement the tracing semantics described in the Dapper
> paper [2] in order to trace the causal relationships between the different
> processing phases that an IO request may trigger. Our target is an end-to-end
> visualisation of the request's route in the system, accompanied by information
> concerning latencies in each processing phase. Thanks to LTTng this can happen
> with a minimal overhead and in realtime. In order to visualize the results we
> have integrated Twitter's Zipkin [3], (which is a tracing system
> entirely based on
> Dapper) with LTTng.
>
> You can find a proof of concept of what we've done so far here:
>
> http://snf-551656.vm.okeanos.grnet.gr:8080/traces/0b554b8a48cb3e84?serviceName=MOSDOp
>
> In the above link you can see the trace of a write request served by a RADOS
> pool with replication level set to 3 (two replicas).
>
> We'd love to have early feedback and comments from you guys too,
> so that we can incorporate useful recommendations. You can find all
> the relevant code
> here[5][6]. If you have any questions or you wish to experiment with the
> project please do not hesitate to contact us.
>
> Kind regards,
> Marios
>
> [1]https://github.com/ceph/ceph/tree/wip-lttng
> [2]http://static.googleusercontent.com/media/research.google.com/el//pubs/archive/36356.pdf
> [3]http://twitter.github.io/zipkin/
> [4] https://github.com/marioskogias/blkin
> [5] https://github.com/marioskogias/babeltrace-plugins
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ιnstrumenting RADOS with Zipkin + LTTng

[Samuel Just]

I'm probably missing something, is there supposed to be a [6] with
your ceph changes?
-Sam

On Fri, Aug 1, 2014 at 1:17 PM, Samuel Just <sam.just@inktank.com> wrote:
> Reading!
> -Sam
>
> On Fri, Aug 1, 2014 at 9:28 AM, Marios-Evaggelos Kogias
> <marioskogias@gmail.com> wrote:
>> Hello all,
>>
>> my name is Marios Kogias and I am a student at the National Technical
>> University of Athens. As part of my diploma thesis and my participation in
>> Google Summer of Code 2014 (in the LTTng organization) I am working on a
>> low-overhead tracing infrastructure for distributed systems. I am also
>> collaborating with the Synnefo team (https://www.synnefo.org/) and especially
>> with Vangelis Koukis, Constantinos Venetsanopoulos and Filippos Giannakos (cc)
>>
>> Some time ago, we started experimenting with RADOS instrumentation
>> using LTTng and
>> we noticed that there are similar endeavours in the Ceph github repository [1].
>>
>> However, unlike your approach, we are following an annotation-based tracing
>> schema, which enables us to track a specific request from the time it enters
>> the system at higher levels till it is finally served by RADOS.
>>
>> In general, we try to implement the tracing semantics described in the Dapper
>> paper [2] in order to trace the causal relationships between the different
>> processing phases that an IO request may trigger. Our target is an end-to-end
>> visualisation of the request's route in the system, accompanied by information
>> concerning latencies in each processing phase. Thanks to LTTng this can happen
>> with a minimal overhead and in realtime. In order to visualize the results we
>> have integrated Twitter's Zipkin [3], (which is a tracing system
>> entirely based on
>> Dapper) with LTTng.
>>
>> You can find a proof of concept of what we've done so far here:
>>
>> http://snf-551656.vm.okeanos.grnet.gr:8080/traces/0b554b8a48cb3e84?serviceName=MOSDOp
>>
>> In the above link you can see the trace of a write request served by a RADOS
>> pool with replication level set to 3 (two replicas).
>>
>> We'd love to have early feedback and comments from you guys too,
>> so that we can incorporate useful recommendations. You can find all
>> the relevant code
>> here[5][6]. If you have any questions or you wish to experiment with the
>> project please do not hesitate to contact us.
>>
>> Kind regards,
>> Marios
>>
>> [1]https://github.com/ceph/ceph/tree/wip-lttng
>> [2]http://static.googleusercontent.com/media/research.google.com/el//pubs/archive/36356.pdf
>> [3]http://twitter.github.io/zipkin/
>> [4] https://github.com/marioskogias/blkin
>> [5] https://github.com/marioskogias/babeltrace-plugins
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ιnstrumenting RADOS with Zipkin + LTTng

[Marios-Evaggelos Kogias]

2014-08-02 0:00 GMT+03:00 Adam Crume <adamcrume@gmail.com>:
> I've been assuming the code is at
> https://github.com/marioskogias/ceph/tree/feature-blkin, in which case
> the diff is attached.  I computed the diff using:
> git diff $(git merge-base marioskogias/feature-blkin ceph/master)
> marioskogias/feature-blkin
>

Exactly, this is the repo with the Ceph code changes.
Indeed we updated it so as it has the latest code producing the output shown in
the screenshot which was attached in the previous mail.

Marios

> On Fri, Aug 1, 2014 at 1:24 PM, Samuel Just <sam.just@inktank.com> wrote:
>> I'm probably missing something, is there supposed to be a [6] with
>> your ceph changes?
>> -Sam
>>
>> On Fri, Aug 1, 2014 at 1:17 PM, Samuel Just <sam.just@inktank.com> wrote:
>>> Reading!
>>> -Sam
>>>
>>> On Fri, Aug 1, 2014 at 9:28 AM, Marios-Evaggelos Kogias
>>> <marioskogias@gmail.com> wrote:
>>>> Hello all,
>>>>
>>>> my name is Marios Kogias and I am a student at the National Technical
>>>> University of Athens. As part of my diploma thesis and my participation in
>>>> Google Summer of Code 2014 (in the LTTng organization) I am working on a
>>>> low-overhead tracing infrastructure for distributed systems. I am also
>>>> collaborating with the Synnefo team (https://www.synnefo.org/) and especially
>>>> with Vangelis Koukis, Constantinos Venetsanopoulos and Filippos Giannakos (cc)
>>>>
>>>> Some time ago, we started experimenting with RADOS instrumentation
>>>> using LTTng and
>>>> we noticed that there are similar endeavours in the Ceph github repository [1].
>>>>
>>>> However, unlike your approach, we are following an annotation-based tracing
>>>> schema, which enables us to track a specific request from the time it enters
>>>> the system at higher levels till it is finally served by RADOS.
>>>>
>>>> In general, we try to implement the tracing semantics described in the Dapper
>>>> paper [2] in order to trace the causal relationships between the different
>>>> processing phases that an IO request may trigger. Our target is an end-to-end
>>>> visualisation of the request's route in the system, accompanied by information
>>>> concerning latencies in each processing phase. Thanks to LTTng this can happen
>>>> with a minimal overhead and in realtime. In order to visualize the results we
>>>> have integrated Twitter's Zipkin [3], (which is a tracing system
>>>> entirely based on
>>>> Dapper) with LTTng.
>>>>
>>>> You can find a proof of concept of what we've done so far here:
>>>>
>>>> http://snf-551656.vm.okeanos.grnet.gr:8080/traces/0b554b8a48cb3e84?serviceName=MOSDOp
>>>>
>>>> In the above link you can see the trace of a write request served by a RADOS
>>>> pool with replication level set to 3 (two replicas).
>>>>
>>>> We'd love to have early feedback and comments from you guys too,
>>>> so that we can incorporate useful recommendations. You can find all
>>>> the relevant code
>>>> here[5][6]. If you have any questions or you wish to experiment with the
>>>> project please do not hesitate to contact us.
>>>>
>>>> Kind regards,
>>>> Marios
>>>>
>>>> [1]https://github.com/ceph/ceph/tree/wip-lttng
>>>> [2]http://static.googleusercontent.com/media/research.google.com/el//pubs/archive/36356.pdf
>>>> [3]http://twitter.github.io/zipkin/
>>>> [4] https://github.com/marioskogias/blkin
>>>> [5] https://github.com/marioskogias/babeltrace-plugins
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ιnstrumenting RADOS with Zipkin + LTTng

[Samuel Just]

Would you be interested in trying to submit a cleaned up version of
the branch as a pull request?  That would make it easier to get it
into a merge-able state.
-Sam

On Tue, Aug 5, 2014 at 9:22 AM, Marios-Evaggelos Kogias
<marioskogias@gmail.com> wrote:
> 2014-08-02 0:00 GMT+03:00 Adam Crume <adamcrume@gmail.com>:
>> I've been assuming the code is at
>> https://github.com/marioskogias/ceph/tree/feature-blkin, in which case
>> the diff is attached.  I computed the diff using:
>> git diff $(git merge-base marioskogias/feature-blkin ceph/master)
>> marioskogias/feature-blkin
>>
>
> Exactly, this is the repo with the Ceph code changes.
> Indeed we updated it so as it has the latest code producing the output shown in
> the screenshot which was attached in the previous mail.
>
> Marios
>
>> On Fri, Aug 1, 2014 at 1:24 PM, Samuel Just <sam.just@inktank.com> wrote:
>>> I'm probably missing something, is there supposed to be a [6] with
>>> your ceph changes?
>>> -Sam
>>>
>>> On Fri, Aug 1, 2014 at 1:17 PM, Samuel Just <sam.just@inktank.com> wrote:
>>>> Reading!
>>>> -Sam
>>>>
>>>> On Fri, Aug 1, 2014 at 9:28 AM, Marios-Evaggelos Kogias
>>>> <marioskogias@gmail.com> wrote:
>>>>> Hello all,
>>>>>
>>>>> my name is Marios Kogias and I am a student at the National Technical
>>>>> University of Athens. As part of my diploma thesis and my participation in
>>>>> Google Summer of Code 2014 (in the LTTng organization) I am working on a
>>>>> low-overhead tracing infrastructure for distributed systems. I am also
>>>>> collaborating with the Synnefo team (https://www.synnefo.org/) and especially
>>>>> with Vangelis Koukis, Constantinos Venetsanopoulos and Filippos Giannakos (cc)
>>>>>
>>>>> Some time ago, we started experimenting with RADOS instrumentation
>>>>> using LTTng and
>>>>> we noticed that there are similar endeavours in the Ceph github repository [1].
>>>>>
>>>>> However, unlike your approach, we are following an annotation-based tracing
>>>>> schema, which enables us to track a specific request from the time it enters
>>>>> the system at higher levels till it is finally served by RADOS.
>>>>>
>>>>> In general, we try to implement the tracing semantics described in the Dapper
>>>>> paper [2] in order to trace the causal relationships between the different
>>>>> processing phases that an IO request may trigger. Our target is an end-to-end
>>>>> visualisation of the request's route in the system, accompanied by information
>>>>> concerning latencies in each processing phase. Thanks to LTTng this can happen
>>>>> with a minimal overhead and in realtime. In order to visualize the results we
>>>>> have integrated Twitter's Zipkin [3], (which is a tracing system
>>>>> entirely based on
>>>>> Dapper) with LTTng.
>>>>>
>>>>> You can find a proof of concept of what we've done so far here:
>>>>>
>>>>> http://snf-551656.vm.okeanos.grnet.gr:8080/traces/0b554b8a48cb3e84?serviceName=MOSDOp
>>>>>
>>>>> In the above link you can see the trace of a write request served by a RADOS
>>>>> pool with replication level set to 3 (two replicas).
>>>>>
>>>>> We'd love to have early feedback and comments from you guys too,
>>>>> so that we can incorporate useful recommendations. You can find all
>>>>> the relevant code
>>>>> here[5][6]. If you have any questions or you wish to experiment with the
>>>>> project please do not hesitate to contact us.
>>>>>
>>>>> Kind regards,
>>>>> Marios
>>>>>
>>>>> [1]https://github.com/ceph/ceph/tree/wip-lttng
>>>>> [2]http://static.googleusercontent.com/media/research.google.com/el//pubs/archive/36356.pdf
>>>>> [3]http://twitter.github.io/zipkin/
>>>>> [4] https://github.com/marioskogias/blkin
>>>>> [5] https://github.com/marioskogias/babeltrace-plugins
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ιnstrumenting RADOS with Zipkin + LTTng

[Adam Crume]

I'm a developer working on RBD replay, so I've written a lot of the
tracing code.  I'd like to start out by saying that I'm speaking for
myself, not for the Ceph project as a whole.

This certainly is interesting.  This would be useful for analysis that
simple statistics couldn't capture, like correlations between
latencies of different components.  It would be even more interesting
with more layers, e.g. including RGW, RBD, or CephFS.

We do have different goals in tracing.  Your work (as I understand it)
is intended to help understand performance, in which case it makes
sense to capture details about suboperations.  Our work is intended to
capture a workload so that it can be replayed.  For workload capture,
we need a different set of details, such as the object affected,
request parameters, and so on.  There's likely to be a good amount of
overlap, though.  The tracing required for workload capture might even
be a subset of that useful for performance analysis.

It seems like separating reads and writes would be a huge benefit,
since they have very different behavior and performance.  Capturing
data size would be helpful, too.

By the way, that Zipkin UI is pretty slick.  Nice choice.

Adam

On Fri, Aug 1, 2014 at 9:28 AM, Marios-Evaggelos Kogias
<marioskogias@gmail.com> wrote:
> Hello all,
>
> my name is Marios Kogias and I am a student at the National Technical
> University of Athens. As part of my diploma thesis and my participation in
> Google Summer of Code 2014 (in the LTTng organization) I am working on a
> low-overhead tracing infrastructure for distributed systems. I am also
> collaborating with the Synnefo team (https://www.synnefo.org/) and especially
> with Vangelis Koukis, Constantinos Venetsanopoulos and Filippos Giannakos (cc)
>
> Some time ago, we started experimenting with RADOS instrumentation
> using LTTng and
> we noticed that there are similar endeavours in the Ceph github repository [1].
>
> However, unlike your approach, we are following an annotation-based tracing
> schema, which enables us to track a specific request from the time it enters
> the system at higher levels till it is finally served by RADOS.
>
> In general, we try to implement the tracing semantics described in the Dapper
> paper [2] in order to trace the causal relationships between the different
> processing phases that an IO request may trigger. Our target is an end-to-end
> visualisation of the request's route in the system, accompanied by information
> concerning latencies in each processing phase. Thanks to LTTng this can happen
> with a minimal overhead and in realtime. In order to visualize the results we
> have integrated Twitter's Zipkin [3], (which is a tracing system
> entirely based on
> Dapper) with LTTng.
>
> You can find a proof of concept of what we've done so far here:
>
> http://snf-551656.vm.okeanos.grnet.gr:8080/traces/0b554b8a48cb3e84?serviceName=MOSDOp
>
> In the above link you can see the trace of a write request served by a RADOS
> pool with replication level set to 3 (two replicas).
>
> We'd love to have early feedback and comments from you guys too,
> so that we can incorporate useful recommendations. You can find all
> the relevant code
> here[5][6]. If you have any questions or you wish to experiment with the
> project please do not hesitate to contact us.
>
> Kind regards,
> Marios
>
> [1]https://github.com/ceph/ceph/tree/wip-lttng
> [2]http://static.googleusercontent.com/media/research.google.com/el//pubs/archive/36356.pdf
> [3]http://twitter.github.io/zipkin/
> [4] https://github.com/marioskogias/blkin
> [5] https://github.com/marioskogias/babeltrace-plugins
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ιnstrumenting RADOS with Zipkin + LTTng

[Marios-Evaggelos Kogias]

Hello Adam,

>I'm a developer working on RBD replay, so I've written a lot of the
>tracing code.  I'd like to start out by saying that I'm speaking for
>myself, not for the Ceph project as a whole.
>
>This certainly is interesting.  This would be useful for analysis that
>simple statistics couldn't capture, like correlations between
>latencies of different components.  It would be even more interesting
>with more layers, e.g. including RGW, RBD, or CephFS.
>

Your are absolutely right. Tracing requests across different storage
layers was the main reason for creating this infrastructure. We are
using RADOS as a storage backend for Archipelago [1], our custom
storage layer similar to RBD and RGW and as you can see here

http://snf-541212.vm.okeanos.grnet.gr:8080/traces/1a5a85354acb719c?serviceName=bench

we can trace the requests from their creation in the upper layers till
they are finally served by RADOS. In the above link 'bench', 'vlmc',
'mapper' and 'radosd' are Archipelago layers. 'radosd' is the layer that
uses librados and  communicates with the RADOS backend. Of course,
one could replace Archipelago with RGW, RBD, CephFS or any other software
layer on top of librados.

>We do have different goals in tracing.  Your work (as I understand it)
>is intended to help understand performance, in which case it makes
>sense to capture details about suboperations.  Our work is intended to
>capture a workload so that it can be replayed.  For workload capture,
>we need a different set of details, such as the object affected,
>request parameters, and so on.  There's likely to be a good amount of
>overlap, though.  The tracing required for workload capture might even
>be a subset of that useful for performance analysis.

Although our main goal was indeed live, cross-layer performance tracing,
it now seems that this infrastructure can cover more generic needs as
well. According to Dapper logic each traced information (annotation)
should belong to a group of annotations (span). There are two different
kinds of annotations, timestamp and key-value. Timestamp annotations are
used to trace specific events, while key-value annotations are used to
log extra trace information. So, the information you are asking for
could be traced as a key-value annotation, e.g. "Operation"="read".
To demonstrate the above, you can take a look at the key-value
annotations of the "Handling op" span of the primary osd (osd.0) by
clicking on it. There, we have logged the object that is affected by
the traced request. We could do the same thing with the information
needed for the workload to be replayed and filter by any criteria
desired, like read or write requests for example.

In addition, using this methodology saves us from manually correlating
the traces, since they are already grouped in parent-child relations. Then
we can access them easily either from the Zipkin UI or from the SQL-interface.

>
>It seems like separating reads and writes would be a huge benefit,
>since they have very different behavior and performance.  Capturing
>data size would be helpful, too.

As mentioned above, this information can be captured as key-value
annotations and can be easily filtered either from the Zipkin UI
or using the SQL-interface. And yes, we should definitely add the
data size too, it's rather trivial.

>
>By the way, that Zipkin UI is pretty slick.  Nice choice.

As you noted, the traces you require for the workload capturing seem to
be a subset of what this infrastructure can trace, so it would be
great if we dug this further and see if we can even join efforts on
this. Combining your RADOS expertise with our work with the LTTng
community and Zipkin could probably result in something really
interesting. What do you think?

Comments from the rest of the Ceph community are also welcome, of course.

Thanks a lot for your feedback,
Marios

[1] https://www.usenix.org/system/files/login/articles/02_giannakos.pdf

2014-08-01 23:54 GMT+03:00 Adam Crume <adamcrume@gmail.com>:
> I'm a developer working on RBD replay, so I've written a lot of the
> tracing code.  I'd like to start out by saying that I'm speaking for
> myself, not for the Ceph project as a whole.
>
> This certainly is interesting.  This would be useful for analysis that
> simple statistics couldn't capture, like correlations between
> latencies of different components.  It would be even more interesting
> with more layers, e.g. including RGW, RBD, or CephFS.
>
> We do have different goals in tracing.  Your work (as I understand it)
> is intended to help understand performance, in which case it makes
> sense to capture details about suboperations.  Our work is intended to
> capture a workload so that it can be replayed.  For workload capture,
> we need a different set of details, such as the object affected,
> request parameters, and so on.  There's likely to be a good amount of
> overlap, though.  The tracing required for workload capture might even
> be a subset of that useful for performance analysis.
>
> It seems like separating reads and writes would be a huge benefit,
> since they have very different behavior and performance.  Capturing
> data size would be helpful, too.
>
> By the way, that Zipkin UI is pretty slick.  Nice choice.
>
> Adam
>
> On Fri, Aug 1, 2014 at 9:28 AM, Marios-Evaggelos Kogias
> <marioskogias@gmail.com> wrote:
>> Hello all,
>>
>> my name is Marios Kogias and I am a student at the National Technical
>> University of Athens. As part of my diploma thesis and my participation in
>> Google Summer of Code 2014 (in the LTTng organization) I am working on a
>> low-overhead tracing infrastructure for distributed systems. I am also
>> collaborating with the Synnefo team (https://www.synnefo.org/) and especially
>> with Vangelis Koukis, Constantinos Venetsanopoulos and Filippos Giannakos (cc)
>>
>> Some time ago, we started experimenting with RADOS instrumentation
>> using LTTng and
>> we noticed that there are similar endeavours in the Ceph github repository [1].
>>
>> However, unlike your approach, we are following an annotation-based tracing
>> schema, which enables us to track a specific request from the time it enters
>> the system at higher levels till it is finally served by RADOS.
>>
>> In general, we try to implement the tracing semantics described in the Dapper
>> paper [2] in order to trace the causal relationships between the different
>> processing phases that an IO request may trigger. Our target is an end-to-end
>> visualisation of the request's route in the system, accompanied by information
>> concerning latencies in each processing phase. Thanks to LTTng this can happen
>> with a minimal overhead and in realtime. In order to visualize the results we
>> have integrated Twitter's Zipkin [3], (which is a tracing system
>> entirely based on
>> Dapper) with LTTng.
>>
>> You can find a proof of concept of what we've done so far here:
>>
>> http://snf-551656.vm.okeanos.grnet.gr:8080/traces/0b554b8a48cb3e84?serviceName=MOSDOp
>>
>> In the above link you can see the trace of a write request served by a RADOS
>> pool with replication level set to 3 (two replicas).
>>
>> We'd love to have early feedback and comments from you guys too,
>> so that we can incorporate useful recommendations. You can find all
>> the relevant code
>> here[5][6]. If you have any questions or you wish to experiment with the
>> project please do not hesitate to contact us.
>>
>> Kind regards,
>> Marios
>>
>> [1]https://github.com/ceph/ceph/tree/wip-lttng
>> [2]http://static.googleusercontent.com/media/research.google.com/el//pubs/archive/36356.pdf
>> [3]http://twitter.github.io/zipkin/
>> [4] https://github.com/marioskogias/blkin
>> [5] https://github.com/marioskogias/babeltrace-plugins
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Ιnstrumenting RADOS with Zipkin + LTTng

[Marios-Evaggelos Kogias]

[-- Attachment #1.1: Type: text/plain, Size: 2261 bytes --]

Hello all,

my name is Marios Kogias and I am a student at the National Technical
University of Athens. As part of my diploma thesis and my participation in
Google Summer of Code 2014 (in the LTTng organization) I am working on a
low-overhead tracing infrastructure for distributed systems. I am also
collaborating with the Synnefo team (https://www.synnefo.org/) and
especially
with Vangelis Koukis, Constantinos Venetsanopoulos and Filippos Giannakos
(cc)

Some time ago, we started experimenting with RADOS instrumentation using
LTTng and
we noticed that there are similar endeavours in the Ceph github repository
[1].

However, unlike your approach, we are following an annotation-based tracing
schema, which enables us to track a specific request from the time it enters
the system at higher levels till it is finally served by RADOS.

In general, we try to implement the tracing semantics described in the
Dapper
paper [2] in order to trace the causal relationships between the different
processing phases that an IO request may trigger. Our target is an
end-to-end
visualisation of the request's route in the system, accompanied by
information
concerning latencies in each processing phase. Thanks to LTTng this can
happen
with a minimal overhead and in realtime. In order to visualize the results
we
have integrated Twitter's Zipkin [3], (which is a tracing system entirely
based on
Dapper) with LTTng.

You can find a proof of concept of what we've done so far here:

http://snf-551656.vm.okeanos.grnet.gr:8080/traces/0b554b8a48cb3e84?serviceName=MOSDOp

In the above link you can see the trace of a write request served by a RADOS
pool with replication level set to 3 (two replicas).

We'd love to have early feedback and comments from you guys too,
so that we can incorporate useful recommendations. You can find all the
relevant code
here[5][6]. If you have any questions or you wish to experiment with the
project please do not hesitate to contact us.

Kind regards,
Marios

[1]https://github.com/ceph/ceph/tree/wip-lttng
[2]
http://static.googleusercontent.com/media/research.google.com/el//pubs/archive/36356.pdf
[3]http://twitter.github.io/zipkin/
[4] https://github.com/marioskogias/blkin
[5] https://github.com/marioskogias/babeltrace-plugins

[-- Attachment #1.2: Type: text/html, Size: 3490 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com