Re: [lttng-dev] Java code for CTF trace writing?

Re: [lttng-dev] Java code for CTF trace writing?

[Matthew Khouzam]

Hi all,
Just to mention, everything Philippe said was 100% true, we just right
now are too stretched to add the trace writer to eclipse. It will be
integrated fairly soon, I think.
Matt


On 13-04-15 06:01 PM, Philippe Proulx wrote:
> Actually, that was the exact purpose of my internship at Ericsson last summer.
>
> I designed a new architecture for the CTF part of TMF, but it is still
> not merged/integrated with mainline TMF (I don't even know if it's
> still planned). See
> <http://git.dorsal.polymtl.ca/~pproulx?p=javeltrace.git;a=summary>.
>
> This new architecture is able to read _and write_ CTF packets in a
> generic way. The "Javeltrace" name is just a portmanteau of Java and
> Babeltrace, although should this refined library be integrated into
> TMF, it's not planned to be named like this. However, there exist a
> command-line tool that I made which is officially named Javeltrace and
> uses the aforementioned library. It is described here:
> <http://www.dorsal.polymtl.ca/~pproulx/javeltrace/>. As explained on
> the webpage, the main goal of this utility is to test my work
> interactively.
>
> Keep in mind that everything mentioned here is not thoroughly tested
> and for sure there are a few bugs remaining. Also, the code didn't
> evolve with the latest CTF versions, so there must be incompatibility
> at some level.
>
> The main use case at Ericsson was to synthesize a precise fake trace
> from scratch using a human readable input format that could be
> versioned. The generated CTF traces would then be used to exercise
> parts of TMF to test specific behaviours without having to produce an
> actual real trace. After a few discussions, we chose JSON as an
> interchange format. So you will see lots of JSON related code out
> there. The command-line Javeltrace utility is able to translate
> from/to binary CTF.
>
> At the end of my internship, I started writing docs for what I did.
> It's here: <http://wiki.eclipse.org/Linux_Tools_Project/TMF/CTF_guide>.
> It's not finished, but almost, so it should be up-to-date with my Git
> codebase.
>
> I also made this as a proof of concept:
> <http://git.dorsal.polymtl.ca/~pproulx?p=javeltrace.git;a=tree;f=lttng/org.eclipse.linuxtools.ctf.core/src/org/eclipse/linuxtools/ctf/core/trace/out;h=67ab3097e9f3fc52a685d29620e4c40ee26ee896;hb=47b4cef68eb4524fc9e1865f58474e0bc81eba77>
> (see all Mug*.java files). MugTracer is a simple Java tracer that
> produces native CTF without even using the rest of my library since
> CTF is so easy to *write*. It has a consumer thread and worked well,
> although I didn't run any benchmark and I believe it's really slow
> compared to UST.
>
> Feel free to contact me, should you have any question about this.
>
> On 15 April 2013 17:14, Aaron Spear <aspear-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org> wrote:
>> Hi all,
>>
>> I was wondering if anyone knew of some open source Java library that could WRITE CTF traces? I am using the linuxtools/TMF plugins to read CTF traces, but now need to write them from Java as well.
>>
>> I have a use case where I have an "event bus" in the Java app world and I would like to persist this event stream as a CTF trace. In this particular use case, the speed/low intrusiveness of LTTng UST is not as important as the portability, so a pure Java solution is ideal, though not strictly required.
>>
>> Also, please let me know if there are others out there who are interested in collaborating in writing such a library.
>>
>> regards,
>> Aaron Spear
>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org
>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> _______________________________________________
> lttng-dev mailing list
> lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Java code for CTF trace writing?

[Aaron Spear]

----- Original Message -----
> Hi all,
> Just to mention, everything Philippe said was 100% true, we just right
> now are too stretched to add the trace writer to eclipse. It will be
> integrated fairly soon, I think.
> Matt

Hi Matthew,

Sounds very good, do you have a ball park estimate when "fairly soon" is?  

Also, who is planning to do the work?  I would be happy to assist if I can.  

As I at least implied, I am happy to collaborate on this.  Right now my prototype is writing out events to a text log file, which is vastly less than ideal.  In short order I am going to need to be able to do two things:

1) write CTF traces from Java code
2) be able to append to this trace dynamically while it is being analyzed.  Ideally the view registers a listener with the model and perhaps also specifying some hysteresis (so it is notified on a block of events, not every event), and then when it is notified it can update (optionally subject to an "update policy" selected in the view, as is often done with debugging)

cheers,
Aaron

Re: [lttng-dev] Java code for CTF trace writing?

[Matthew Khouzam]

On 13-04-23 02:21 PM, Aaron Spear wrote:
>
> ----- Original Message -----
>> Hi all,
>> Just to mention, everything Philippe said was 100% true, we just right
>> now are too stretched to add the trace writer to eclipse. It will be
>> integrated fairly soon, I think.
>> Matt
> Hi Matthew,
>
> Sounds very good, do you have a ball park estimate when "fairly soon" is?  
I will probably start after the feature freeze in eclipse. June sounds
+- right.
>
> Also, who is planning to do the work?  I would be happy to assist if I can.  
Me probably. ;)
>
> As I at least implied, I am happy to collaborate on this.  Right now my prototype is writing out events to a text log file, which is vastly less than ideal.  In short order I am going to need to be able to do two things:
>
> 1) write CTF traces from Java code
Same objective here.
> 2) be able to append to this trace dynamically while it is being analyzed.  Ideally the view registers a listener with the model and perhaps also specifying some hysteresis (so it is notified on a block of events, not every event), and then when it is notified it can update (optionally subject to an "update policy" selected in the view, as is often done with debugging)
We don't support "Live" trace reading yet. I'm not saying it won't work,
but it has not been tested.

When would you be available to work on this feature, I would love to
sync up our efforts for this.
>
> cheers,
> Aaron

Re: [lttng-dev] Java code for CTF trace writing?

[Aaron Spear]

> When would you be available to work on this feature, I would love to
> sync up our efforts for this.

The way things look right now, I will have bandwidth for it in another month, so middle to end of May. Perhaps sooner if all the stars align.

Aaron

Re: Java code for CTF trace writing?

[Ostermueller, Erik]

Hello all,

I've got a question about the Java code that writes CTF traces.
Would this design provide a single trace with Java events and OS events in the same trace?
Here is an example:

Consider an all-Java program that sends an XML requests request over HTTP.
Would like the trace detail to show Java method activity as well as OS activity that opens/closes the socket.

This could be very helpful for unveiling the details of how Java code consumes hardware resources.

Thanks,
--Erik

-----Original Message-----
From: Aaron Spear [mailto:aspear@vmware.com] 
Sent: Tuesday, April 23, 2013 3:51 PM
To: Matthew Khouzam
Cc: Philippe Proulx; Dominique Toupin; lttng-dev@lists.lttng.org; Linux Tools developer discussions
Subject: Re: [lttng-dev] Java code for CTF trace writing?

> When would you be available to work on this feature, I would love to 
> sync up our efforts for this.

The way things look right now, I will have bandwidth for it in another month, so middle to end of May. Perhaps sooner if all the stars align.

Aaron

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.

Re: Java code for CTF trace writing?

[Philippe Proulx]

[-- Attachment #1.1: Type: text/plain, Size: 2196 bytes --]

I don't think so, but usually you would have a trace for your Java events
and another one for OS events that are recorded simultaneously.

Then you simply open both with Babeltrace or your favorite trace viewer and
you should see interleaving events.

You might have problems with events timing synchronization though if you
plan to produce your own CTF trace.

Philippe Proulx


On Mon, May 6, 2013 at 10:33 AM, Ostermueller, Erik <
Erik.Ostermueller@fisglobal.com> wrote:

> Hello all,
>
> I've got a question about the Java code that writes CTF traces.
> Would this design provide a single trace with Java events and OS events in
> the same trace?
> Here is an example:
>
> Consider an all-Java program that sends an XML requests request over HTTP.
> Would like the trace detail to show Java method activity as well as OS
> activity that opens/closes the socket.
>
> This could be very helpful for unveiling the details of how Java code
> consumes hardware resources.
>
> Thanks,
> --Erik
>
> -----Original Message-----
> From: Aaron Spear [mailto:aspear@vmware.com]
> Sent: Tuesday, April 23, 2013 3:51 PM
> To: Matthew Khouzam
> Cc: Philippe Proulx; Dominique Toupin; lttng-dev@lists.lttng.org; Linux
> Tools developer discussions
> Subject: Re: [lttng-dev] Java code for CTF trace writing?
>
> > When would you be available to work on this feature, I would love to
> > sync up our efforts for this.
>
> The way things look right now, I will have bandwidth for it in another
> month, so middle to end of May. Perhaps sooner if all the stars align.
>
> Aaron
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
> _____________
> The information contained in this message is proprietary and/or
> confidential. If you are not the intended recipient, please: (i) delete the
> message and all copies; (ii) do not disclose, distribute or use the message
> in any manner; and (iii) notify the sender immediately. In addition, please
> be aware that any message addressed to our domain is subject to archiving
> and review by persons other than the intended recipient. Thank you.
>
>

[-- Attachment #1.2: Type: text/html, Size: 2996 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Java code for CTF trace writing?

[Aaron Spear]

Hi Erik,

The answer at this moment is no, you can't see that.  That said, that is certainly the vision of what I want to create.  I am working on extending a view so that I can see Java methods in one trace alongside state changes that come from another trace. The idea is to have a single view that knows how to present state vs. time and then providers for that state for different kinds of tracing.  I have in fact done exactly what you want (java app trace + linux kernel trace), but it is clunky right now because you cannot see the Linux kernel trace and the Java trace at the same time.

The issue here is simply one of implementation choices related to how traces are viewed.  At this moment the views in the LTTng viewer are singletons setup to only allow one trace to be viewed at a time (mostly.  the "Time Chart" view is an exception to this). What is needed is one of two things (I think both):

1) support for being able to open as many instances of views as you wish, and then "pin" them to a particular trace.  If you could do this then you could open an LTTng linux kernel view and a java trace and whatever else you wanted at the same time. In addition to the pin support the views would need some mechanism for time sync between them (like the current control flow view syncs based on the selection in Histogram view)

2) Change the "state flow view" that I created (borrowing code from the control flow view) so it supports multiple traces (e.g. LTTng kernel trace).  I am working on this right now for data driven traces.  One thought is that this view infrastructure could be changed so that a single view could show all of these different types of traces.  A bunch of refactoring would be required to do this though.  The current ControlFlowView has things very specific to kernel traces that don't make sense generically (e.g. process related details as columns in the trace)

I can't comment as to the official direction of the project (since I am not a committer...) but Perhaps Alexandre or others could comment on thoughts on what could be done to address this and when.

regards, 
Aaron Spear



----- Original Message -----
> Hello all,
> 
> I've got a question about the Java code that writes CTF traces.
> Would this design provide a single trace with Java events and OS events in
> the same trace?
> Here is an example:
> 
> Consider an all-Java program that sends an XML requests request over HTTP.
> Would like the trace detail to show Java method activity as well as OS
> activity that opens/closes the socket.
> 
> This could be very helpful for unveiling the details of how Java code
> consumes hardware resources.
> 
> Thanks,
> --Erik
> 
> -----Original Message-----
> From: Aaron Spear [mailto:aspear-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org]
> Sent: Tuesday, April 23, 2013 3:51 PM
> To: Matthew Khouzam
> Cc: Philippe Proulx; Dominique Toupin; lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org; Linux Tools
> developer discussions
> Subject: Re: [lttng-dev] Java code for CTF trace writing?
> 
> > When would you be available to work on this feature, I would love to
> > sync up our efforts for this.
> 
> The way things look right now, I will have bandwidth for it in another month,
> so middle to end of May. Perhaps sooner if all the stars align.
> 
> Aaron
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 
> _____________
> The information contained in this message is proprietary and/or confidential.
> If you are not the intended recipient, please: (i) delete the message and
> all copies; (ii) do not disclose, distribute or use the message in any
> manner; and (iii) notify the sender immediately. In addition, please be
> aware that any message addressed to our domain is subject to archiving and
> review by persons other than the intended recipient. Thank you
> 

Re: Java code for CTF trace writing?

[Bernd Hufmann]

Hi Aaron

I'd like to comment on the pinning of views (the rest of the discussion is already covered by Alexandre).

> 1) support for being able to open as many instances of views as you wish, and then "pin" them to a particular trace.  If you could do this then you could open an LTTng linux kernel view and a java trace and 
> whatever else you wanted at the same time. In addition to the pin support the views would need some mechanism for time sync between them (like the current control flow view syncs based on the selection in 
> Histogram view)

Since Eclipse supports to have multiple instances of the same view it should be straight forward to support multiple instances of the Tracing views. To do that we need to provide an unique instance ID to distinguish the different views. The data handled in the view has to be specific to the instance of the view. For our purpose, the data to be displayed will be related to the trace implementation (ITmfTrace). We have to make sure that the data and view are separated properly (data encapsulation). The next question is how to open a new instance of a tracing view (e.g. ControlFlowView). We could implement it similarly to the Eclipse Properties view. This means we have pin button and command to pin the view to the current active trace. To not change content when changing the trace, the pinned
  view has to ignore all relevant signals sent between views (e.g. TmfTraceSelectedSignal, TmfTraceOpenedSignal). Additionally, the views need a "Clone" or "New Instance" command (e.g. in the
  view menu) that creates a new instance of the view. The new view will be filled with the current active trace information (or stays empty initially) and when changing the trace the cloned view will change the content with the new trace information. 

For the time synchronisation between views, we could have multiple instances of the Histogram View, each one synchronises with the corresponding tracing view instances from above. Or maybe the TimeChartView is sufficient to handle the time synchronisation.

Would it be beneficial to have a global action that would allow the user to clone multiple tracing views at the time? This action could allow the user to select the views to be cloned. This could be handy if there are multiple views that need to be cloned at the same time.

By the way, the base view class TmfView has already an action to pin a view that extends TmfView. Once created it will add a pin button (toggle) to the view's toolbar. This action toggles a flag in the base class and all sub-class can query this information and act accordingly. Initially this pin action was intended to be used to pin a view for the current selected time and time range. It hasn't been used by any of the Open Souce Tracing views. Even if it was meant for pinning to the current time, it's free for use and could be also be used for pinning to the current trace.

Best Regards
Bernd

________________________________________
From: Aaron Spear [aspear-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org]
Sent: May 6, 2013 1:53 PM
To: Erik Ostermueller
Cc: Philippe Proulx; Dominique Toupin; Linux Tools developer discussions; lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org
Subject: Re: [lttng-dev] Java code for CTF trace writing?

Hi Erik,

The answer at this moment is no, you can't see that.  That said, that is certainly the vision of what I want to create.  I am working on extending a view so that I can see Java methods in one trace alongside state changes that come from another trace. The idea is to have a single view that knows how to present state vs. time and then providers for that state for different kinds of tracing.  I have in fact done exactly what you want (java app trace + linux kernel trace), but it is clunky right now because you cannot see the Linux kernel trace and the Java trace at the same time.

The issue here is simply one of implementation choices related to how traces are viewed.  At this moment the views in the LTTng viewer are singletons setup to only allow one trace to be viewed at a time (mostly.  the "Time Chart" view is an exception to this). What is needed is one of two things (I think both):

1) support for being able to open as many instances of views as you wish, and then "pin" them to a particular trace.  If you could do this then you could open an LTTng linux kernel view and a java trace and whatever else you wanted at the same time. In addition to the pin support the views would need some mechanism for time sync between them (like the current control flow view syncs based on the selection in Histogram view)

2) Change the "state flow view" that I created (borrowing code from the control flow view) so it supports multiple traces (e.g. LTTng kernel trace).  I am working on this right now for data driven traces.  One thought is that this view infrastructure could be changed so that a single view could show all of these different types of traces.  A bunch of refactoring would be required to do this though.  The current ControlFlowView has things very specific to kernel traces that don't make sense generically (e.g. process related details as columns in the trace)

I can't comment as to the official direction of the project (since I am not a committer...) but Perhaps Alexandre or others could comment on thoughts on what could be done to address this and when.

regards,
Aaron Spear



----- Original Message -----
> Hello all,
>
> I've got a question about the Java code that writes CTF traces.
> Would this design provide a single trace with Java events and OS events in
> the same trace?
> Here is an example:
>
> Consider an all-Java program that sends an XML requests request over HTTP.
> Would like the trace detail to show Java method activity as well as OS
> activity that opens/closes the socket.
>
> This could be very helpful for unveiling the details of how Java code
> consumes hardware resources.
>
> Thanks,
> --Erik
>
> -----Original Message-----
> From: Aaron Spear [mailto:aspear-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org]
> Sent: Tuesday, April 23, 2013 3:51 PM
> To: Matthew Khouzam
> Cc: Philippe Proulx; Dominique Toupin; lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org; Linux Tools
> developer discussions
> Subject: Re: [lttng-dev] Java code for CTF trace writing?
>
> > When would you be available to work on this feature, I would love to
> > sync up our efforts for this.
>
> The way things look right now, I will have bandwidth for it in another month,
> so middle to end of May. Perhaps sooner if all the stars align.
>
> Aaron
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
> _____________
> The information contained in this message is proprietary and/or confidential.
> If you are not the intended recipient, please: (i) delete the message and
> all copies; (ii) do not disclose, distribute or use the message in any
> manner; and (iii) notify the sender immediately. In addition, please be
> aware that any message addressed to our domain is subject to archiving and
> review by persons other than the intended recipient. Thank you
>

_______________________________________________
lttng-dev mailing list
lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: Java code for CTF trace writing?

[Aaron Spear]

Hi Bernd,

Thanks much for the insightful comments.  Some more thoughts injected below:

----- Original Message -----
> Hi Aaron
> 
> I'd like to comment on the pinning of views (the rest of the discussion is
> already covered by Alexandre).
> 
> > 1) support for being able to open as many instances of views as you wish,
> > and then "pin" them to a particular trace.  If you could do this then you
> > could open an LTTng linux kernel view and a java trace and
> > whatever else you wanted at the same time. In addition to the pin support
> > the views would need some mechanism for time sync between them (like the
> > current control flow view syncs based on the selection in
> > Histogram view)
> 
> Since Eclipse supports to have multiple instances of the same view it should
> be straight forward to support multiple instances of the Tracing views. To
> do that we need to provide an unique instance ID to distinguish the
> different views. The data handled in the view has to be specific to the
> instance of the view. For our purpose, the data to be displayed will be
> related to the trace implementation (ITmfTrace). We have to make sure that
> the data and view are separated properly (data encapsulation). The next
> question is how to open a new instance of a tracing view (e.g.
> ControlFlowView). We could implement it similarly to the Eclipse Properties
> view. This means we have pin button and command to pin the view to the
> current active trace. To not change content when changing the trace, the
> pinned view has to ignore all relevant signals sent between views (e.g.
> TmfTraceSelectedSignal, TmfTraceOpenedSignal). Additionally, the views need
> a "Clone" or "New Instance" command (e.g. in the view menu) that creates a
> new instance of the view. The new view will be filled with the current
> active trace information (or stays empty initially) and when changing the
> trace the cloned view will change the content with the new trace
> information.
> 
> For the time synchronisation between views, we could have multiple instances
> of the Histogram View, each one synchronises with the corresponding tracing
> view instances from above. Or maybe the TimeChartView is sufficient to
> handle the time synchronisation.
> 
> Would it be beneficial to have a global action that would allow the user to
> clone multiple tracing views at the time? This action could allow the user
> to select the views to be cloned. This could be handy if there are multiple
> views that need to be cloned at the same time.

One thing that I think is important to keep in mind is that in the average users mind, they will probably not think much about the editor-view model like it is in the Eclipse sense, meaning that for them the control flow, or whatever other view they are looking primarily that shows behavior vs. time is in their mind "the trace" as opposed to the tabular editor with all the raw trace events in it.  I mention it only because whatever it is that we do, it seems to me that it would be nice if the usage of it was fluid and natural.  I could see wanting to a new control flow instance open every time that I open another trace (or having the same view simply add the additional trace content).  So all that to say, I don't think cloning a view, switching to a different trace, and then pinning is the
  only way that we want to have it work.  I think that it would be sensible to have a preference that created a mode whereby any time that you opened a new trace, a separate instance of the v
 iews opened, and those separate instances were setup to listen for "shared time navigation signals" which control what time range and particular time is selected globally. 

Also regarding the current histogram view, this view is really nice in the way that it functions and controls time in the control flow and other views, but I do think it is a screen real estate hog.  I don't think that having multiple instances of this is a good idea.  I have multiple 27 inch monitors, and in my current usage I struggle for space.  So, I cannot help but wonder if there could be another incarnation of it that is more like a combination of the time chart view and the histogram view.  In this updated time navigator, you could define synchronization groups for traces (and have a single global group by default) where for a given group there is a single range selection, and all trace views sync to this range.  It also seems to me that this sort of presentation might also make se
 nse to be the place where you can adjust the time offset/bias on traces manually, or you could have access to plugins that process multiple traces and then do smart time adjustment based on 
 network or other events expressed in CTF to indicate relationships between traces (such as is being discussed in this thread)

> 
> By the way, the base view class TmfView has already an action to pin a view
> that extends TmfView. Once created it will add a pin button (toggle) to the
> view's toolbar. This action toggles a flag in the base class and all
> sub-class can query this information and act accordingly. Initially this pin
> action was intended to be used to pin a view for the current selected time
> and time range. It hasn't been used by any of the Open Souce Tracing views.
> Even if it was meant for pinning to the current time, it's free for use and
> could be also be used for pinning to the current trace.

Sounds like a start.  Seems like having a shared facility on each view so that it can change the trace that it is pinned to and then its update/synchronization behavior would be good.  e.g. a radio button selection/list of the "sync groups" that are available.  If you select one, then any time events are fired to change the selection/range for that group then the view updates.  Otherwise it does not.  Will will likely also need update policies on how fast to update and such in the future when doing live traces.

> 
> Best Regards
> Bernd
> 
> ________________________________________
> From: Aaron Spear [aspear-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org]
> Sent: May 6, 2013 1:53 PM
> To: Erik Ostermueller
> Cc: Philippe Proulx; Dominique Toupin; Linux Tools developer discussions;
> lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org
> Subject: Re: [lttng-dev] Java code for CTF trace writing?
> 
> Hi Erik,
> 
> The answer at this moment is no, you can't see that.  That said, that is
> certainly the vision of what I want to create.  I am working on extending a
> view so that I can see Java methods in one trace alongside state changes
> that come from another trace. The idea is to have a single view that knows
> how to present state vs. time and then providers for that state for
> different kinds of tracing.  I have in fact done exactly what you want (java
> app trace + linux kernel trace), but it is clunky right now because you
> cannot see the Linux kernel trace and the Java trace at the same time.
> 
> The issue here is simply one of implementation choices related to how traces
> are viewed.  At this moment the views in the LTTng viewer are singletons
> setup to only allow one trace to be viewed at a time (mostly.  the "Time
> Chart" view is an exception to this). What is needed is one of two things (I
> think both):
> 
> 1) support for being able to open as many instances of views as you wish, and
> then "pin" them to a particular trace.  If you could do this then you could
> open an LTTng linux kernel view and a java trace and whatever else you
> wanted at the same time. In addition to the pin support the views would need
> some mechanism for time sync between them (like the current control flow
> view syncs based on the selection in Histogram view)
> 
> 2) Change the "state flow view" that I created (borrowing code from the
> control flow view) so it supports multiple traces (e.g. LTTng kernel trace).
> I am working on this right now for data driven traces.  One thought is that
> this view infrastructure could be changed so that a single view could show
> all of these different types of traces.  A bunch of refactoring would be
> required to do this though.  The current ControlFlowView has things very
> specific to kernel traces that don't make sense generically (e.g. process
> related details as columns in the trace)
> 
> I can't comment as to the official direction of the project (since I am not a
> committer...) but Perhaps Alexandre or others could comment on thoughts on
> what could be done to address this and when.
> 
> regards,
> Aaron Spear
> 
> 
> 
> ----- Original Message -----
> > Hello all,
> >
> > I've got a question about the Java code that writes CTF traces.
> > Would this design provide a single trace with Java events and OS events in
> > the same trace?
> > Here is an example:
> >
> > Consider an all-Java program that sends an XML requests request over HTTP.
> > Would like the trace detail to show Java method activity as well as OS
> > activity that opens/closes the socket.
> >
> > This could be very helpful for unveiling the details of how Java code
> > consumes hardware resources.
> >
> > Thanks,
> > --Erik
> >
> > -----Original Message-----
> > From: Aaron Spear [mailto:aspear-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org]
> > Sent: Tuesday, April 23, 2013 3:51 PM
> > To: Matthew Khouzam
> > Cc: Philippe Proulx; Dominique Toupin; lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org; Linux
> > Tools
> > developer discussions
> > Subject: Re: [lttng-dev] Java code for CTF trace writing?
> >
> > > When would you be available to work on this feature, I would love to
> > > sync up our efforts for this.
> >
> > The way things look right now, I will have bandwidth for it in another
> > month,
> > so middle to end of May. Perhaps sooner if all the stars align.
> >
> > Aaron
> >
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org
> > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >
> > _____________
> > The information contained in this message is proprietary and/or
> > confidential.
> > If you are not the intended recipient, please: (i) delete the message and
> > all copies; (ii) do not disclose, distribute or use the message in any
> > manner; and (iii) notify the sender immediately. In addition, please be
> > aware that any message addressed to our domain is subject to archiving and
> > review by persons other than the intended recipient. Thank you
> >
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>

Re: Java code for CTF trace writing?

[Bernd Hufmann]

Hi Aaron,

Some more comments below.

Best Regards
Bernd
________________________________________
From: Aaron Spear [aspear-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org]
Sent: May 8, 2013 10:53 AM
To: Bernd Hufmann
Cc: Erik Ostermueller; Philippe Proulx; Dominique Toupin; Linux Tools developer discussions; lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org
Subject: Re: Java code for CTF trace writing?

Hi Bernd,

Thanks much for the insightful comments.  Some more thoughts injected below:

----- Original Message -----
> > Hi Aaron
> >
> > I'd like to comment on the pinning of views (the rest of the discussion is
> > already covered by Alexandre).
> >
> > > 1) support for being able to open as many instances of views as you wish,
> > > and then "pin" them to a particular trace.  If you could do this then you
> > > could open an LTTng linux kernel view and a java trace and
> > > whatever else you wanted at the same time. In addition to the pin support
> > > the views would need some mechanism for time sync between them (like the
> > > current control flow view syncs based on the selection in
> > > Histogram view)
> >
> > Since Eclipse supports to have multiple instances of the same view it should
> > be straight forward to support multiple instances of the Tracing views. To
> > do that we need to provide an unique instance ID to distinguish the
> > different views. The data handled in the view has to be specific to the
> > instance of the view. For our purpose, the data to be displayed will be
> > related to the trace implementation (ITmfTrace). We have to make sure that
> > the data and view are separated properly (data encapsulation). The next
> > question is how to open a new instance of a tracing view (e.g.
> > ControlFlowView). We could implement it similarly to the Eclipse Properties
> > view. This means we have pin button and command to pin the view to the
> > current active trace. To not change content when changing the trace, the
> > pinned view has to ignore all relevant signals sent between views (e.g.
> > TmfTraceSelectedSignal, TmfTraceOpenedSignal). Additionally, the views need
> > a "Clone" or "New Instance" command (e.g. in the view menu) that creates a
> > new instance of the view. The new view will be filled with the current
> > active trace information (or stays empty initially) and when changing the
> > trace the cloned view will change the content with the new trace
> > information.
> >
> > For the time synchronisation between views, we could have multiple instances
> > of the Histogram View, each one synchronises with the corresponding tracing
> > view instances from above. Or maybe the TimeChartView is sufficient to
> > handle the time synchronisation.
> >
> > Would it be beneficial to have a global action that would allow the user to
> > clone multiple tracing views at the time? This action could allow the user
> > to select the views to be cloned. This could be handy if there are multiple
> > views that need to be cloned at the same time.
> 
> One thing that I think is important to keep in mind is that in the average users mind, they will probably not think much about the editor-view model like it is in the Eclipse sense, meaning that for them the control flow, or whatever other view they are looking primarily that shows 
> behavior vs. time is in their mind "the trace" as opposed to the tabular editor with all the raw trace events in it.  I mention it only because whatever it is that we do, it seems to me that it would be nice if the usage of it was fluid and natural.  I could see wanting to a new control flow 
> instance open every time that I open another trace (or having the same view simply add the additional trace content).  So all that to say, I don't think cloning a view, switching to a different trace, and then pinning is the only way that we want to have it work.  I think that it would be 
> sensible to have a preference that created a mode whereby any time that you opened a new trace, a separate instance of the views opened, and those separate instances were setup to listen for "shared time navigation signals" which control what time range and particular time is 
> selected globally.
I agree, that for all the changes we have to keep usability in mind and it should be as intuitive and easy as possible for the user. The preference idea could be a way to go. We could have a default behaviour, where all views are shared across traces (as it is today). The other behaviour (advanced behaviour ?) could be that a new instance of the tracing views is opened when opening a trace. However, for the second behaviour I think it would be beneficial to select the set of view to be opened. We don't want to open views the user is not interested in and having to close them manually.

> 
> Also regarding the current histogram view, this view is really nice in the way that it functions and controls time in the control flow and other views, but I do think it is a screen real estate hog.  I don't think that having multiple instances of this is a good idea.  I have multiple 27 inch 
> monitors, and in my current usage I struggle for space.  So, I cannot help but wonder if there could be another incarnation of it that is more like a combination of the time chart view and the histogram view.  In this updated time navigator, you could define synchronization groups for 
> traces (and have a single global group by default) where for a given group there is a single range selection, and all trace views sync to this range.  It also seems to me that this sort of presentation might also make sense to be the place where you can adjust the time offset/bias on 
> traces manually, or you could have access to plugins that process multiple traces and then do smart time adjustment based on network or other events expressed in CTF to indicate relationships between traces (such as is being discussed in this thread)

Yeah, screen real estate is a problem. Especially with the addition of more analysis views. I think we need to define proper preferences to be able to group views and define their behaviour (i.e. open new instance every time a trace opens, time synchronization across views).  

> 
>
> By the way, the base view class TmfView has already an action to pin a view
> that extends TmfView. Once created it will add a pin button (toggle) to the
> view's toolbar. This action toggles a flag in the base class and all
> sub-class can query this information and act accordingly. Initially this pin
> action was intended to be used to pin a view for the current selected time
> and time range. It hasn't been used by any of the Open Souce Tracing views.
> Even if it was meant for pinning to the current time, it's free for use and
> could be also be used for pinning to the current trace.

> Sounds like a start.  Seems like having a shared facility on each view so that it can change the trace that it is pinned to and then its update/synchronization behavior would be good.  e.g. a radio button selection/list of the "sync groups" that are available.  If you select one, then any 
> time events are fired to change the selection/range for that group then the view updates.  Otherwise it does not.  Will will likely also need update policies on how fast to update and such in the future when doing live traces.

Again here, we have to define preferences and manage them properly in a central way so that each view can access them and adjust it's behaviour.

>
> Best Regards
> Bernd
>
> ________________________________________
> From: Aaron Spear [aspear-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org]
> Sent: May 6, 2013 1:53 PM
> To: Erik Ostermueller
> Cc: Philippe Proulx; Dominique Toupin; Linux Tools developer discussions;
> lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org
> Subject: Re: [lttng-dev] Java code for CTF trace writing?
>
> Hi Erik,
>
> The answer at this moment is no, you can't see that.  That said, that is
> certainly the vision of what I want to create.  I am working on extending a
> view so that I can see Java methods in one trace alongside state changes
> that come from another trace. The idea is to have a single view that knows
> how to present state vs. time and then providers for that state for
> different kinds of tracing.  I have in fact done exactly what you want (java
> app trace + linux kernel trace), but it is clunky right now because you
> cannot see the Linux kernel trace and the Java trace at the same time.
>
> The issue here is simply one of implementation choices related to how traces
> are viewed.  At this moment the views in the LTTng viewer are singletons
> setup to only allow one trace to be viewed at a time (mostly.  the "Time
> Chart" view is an exception to this). What is needed is one of two things (I
> think both):
>
> 1) support for being able to open as many instances of views as you wish, and
> then "pin" them to a particular trace.  If you could do this then you could
> open an LTTng linux kernel view and a java trace and whatever else you
> wanted at the same time. In addition to the pin support the views would need
> some mechanism for time sync between them (like the current control flow
> view syncs based on the selection in Histogram view)
>
> 2) Change the "state flow view" that I created (borrowing code from the
> control flow view) so it supports multiple traces (e.g. LTTng kernel trace).
> I am working on this right now for data driven traces.  One thought is that
> this view infrastructure could be changed so that a single view could show
> all of these different types of traces.  A bunch of refactoring would be
> required to do this though.  The current ControlFlowView has things very
> specific to kernel traces that don't make sense generically (e.g. process
> related details as columns in the trace)
>
> I can't comment as to the official direction of the project (since I am not a
> committer...) but Perhaps Alexandre or others could comment on thoughts on
> what could be done to address this and when.
>
> regards,
> Aaron Spear
>
>
>
> ----- Original Message -----
> > Hello all,
> >
> > I've got a question about the Java code that writes CTF traces.
> > Would this design provide a single trace with Java events and OS events in
> > the same trace?
> > Here is an example:
> >
> > Consider an all-Java program that sends an XML requests request over HTTP.
> > Would like the trace detail to show Java method activity as well as OS
> > activity that opens/closes the socket.
> >
> > This could be very helpful for unveiling the details of how Java code
> > consumes hardware resources.
> >
> > Thanks,
> > --Erik
> >
> > -----Original Message-----
> > From: Aaron Spear [mailto:aspear-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org]
> > Sent: Tuesday, April 23, 2013 3:51 PM
> > To: Matthew Khouzam
> > Cc: Philippe Proulx; Dominique Toupin; lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org; Linux
> > Tools
> > developer discussions
> > Subject: Re: [lttng-dev] Java code for CTF trace writing?
> >
> > > When would you be available to work on this feature, I would love to
> > > sync up our efforts for this.
> >
> > The way things look right now, I will have bandwidth for it in another
> > month,
> > so middle to end of May. Perhaps sooner if all the stars align.
> >
> > Aaron
> >
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org
> > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> >
> > _____________
> > The information contained in this message is proprietary and/or
> > confidential.
> > If you are not the intended recipient, please: (i) delete the message and
> > all copies; (ii) do not disclose, distribute or use the message in any
> > manner; and (iii) notify the sender immediately. In addition, please be
> > aware that any message addressed to our domain is subject to archiving and
> > review by persons other than the intended recipient. Thank you
> >
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev-bnB2LGs2QVJ+nrgayQ7rhA@public.gmane.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>

Re: Java code for CTF trace writing?

[Philippe Proulx]

Actually, that was the exact purpose of my internship at Ericsson last summer.

I designed a new architecture for the CTF part of TMF, but it is still
not merged/integrated with mainline TMF (I don't even know if it's
still planned). See
<http://git.dorsal.polymtl.ca/~pproulx?p=javeltrace.git;a=summary>.

This new architecture is able to read _and write_ CTF packets in a
generic way. The "Javeltrace" name is just a portmanteau of Java and
Babeltrace, although should this refined library be integrated into
TMF, it's not planned to be named like this. However, there exist a
command-line tool that I made which is officially named Javeltrace and
uses the aforementioned library. It is described here:
<http://www.dorsal.polymtl.ca/~pproulx/javeltrace/>. As explained on
the webpage, the main goal of this utility is to test my work
interactively.

Keep in mind that everything mentioned here is not thoroughly tested
and for sure there are a few bugs remaining. Also, the code didn't
evolve with the latest CTF versions, so there must be incompatibility
at some level.

The main use case at Ericsson was to synthesize a precise fake trace
from scratch using a human readable input format that could be
versioned. The generated CTF traces would then be used to exercise
parts of TMF to test specific behaviours without having to produce an
actual real trace. After a few discussions, we chose JSON as an
interchange format. So you will see lots of JSON related code out
there. The command-line Javeltrace utility is able to translate
from/to binary CTF.

At the end of my internship, I started writing docs for what I did.
It's here: <http://wiki.eclipse.org/Linux_Tools_Project/TMF/CTF_guide>.
It's not finished, but almost, so it should be up-to-date with my Git
codebase.

I also made this as a proof of concept:
<http://git.dorsal.polymtl.ca/~pproulx?p=javeltrace.git;a=tree;f=lttng/org.eclipse.linuxtools.ctf.core/src/org/eclipse/linuxtools/ctf/core/trace/out;h=67ab3097e9f3fc52a685d29620e4c40ee26ee896;hb=47b4cef68eb4524fc9e1865f58474e0bc81eba77>
(see all Mug*.java files). MugTracer is a simple Java tracer that
produces native CTF without even using the rest of my library since
CTF is so easy to *write*. It has a consumer thread and worked well,
although I didn't run any benchmark and I believe it's really slow
compared to UST.

Feel free to contact me, should you have any question about this.

On 15 April 2013 17:14, Aaron Spear <aspear@vmware.com> wrote:
> Hi all,
>
> I was wondering if anyone knew of some open source Java library that could WRITE CTF traces? I am using the linuxtools/TMF plugins to read CTF traces, but now need to write them from Java as well.
>
> I have a use case where I have an "event bus" in the Java app world and I would like to persist this event stream as a CTF trace. In this particular use case, the speed/low intrusiveness of LTTng UST is not as important as the portability, so a pure Java solution is ideal, though not strictly required.
>
> Also, please let me know if there are others out there who are interested in collaborating in writing such a library.
>
> regards,
> Aaron Spear
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Java code for CTF trace writing?

[Aaron Spear]

Hi all, 

I was wondering if anyone knew of some open source Java library that could WRITE CTF traces? I am using the linuxtools/TMF plugins to read CTF traces, but now need to write them from Java as well. 

I have a use case where I have an "event bus" in the Java app world and I would like to persist this event stream as a CTF trace. In this particular use case, the speed/low intrusiveness of LTTng UST is not as important as the portability, so a pure Java solution is ideal, though not strictly required. 

Also, please let me know if there are others out there who are interested in collaborating in writing such a library.

regards, 
Aaron Spear