LTTng

LTTng

[Alawneh, Luay]

[-- Attachment #1.1: Type: text/plain, Size: 587 bytes --]

Hi,

I am reading about LTTng and I would like to know if it can replace Log4c for logging user space events in the production environment. In addition to the method enter/exit events we also use Log4c to log many other events to indicate errors and other useful information.

To summarize, I would like to know if LTTng can be used in a production environment for logging different types of events. We are planning to replace Log4c with another logging framework to improve the performance by reducing the time spent in logging.

Thanks,

Luay Alawneh
Nuance Communications


[-- Attachment #1.2: Type: text/html, Size: 3006 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: LTTng

[David Goulet]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Hi Luay,

I am not aware of any Log4c --> LTTng transition work done up to this
day...

One thing I can say is that user space tracing using LTTng will
certainly improve speed and scalability performance.

The event LOGLEVEL feature is pretty interesting for that where you
can only enable a certain loglevel facility for a trace (exactly like
the syslog facilities). You can refer to the lttng-ust(3) man page for
more information.

Moreover, you can also couple kernel traces to your user space traces
which can give you way more information :).

I guess that if now one replies to this thread describing their
experience with log4c vs lttng, you guys will be the first to try a
transition and we will be more than happy to help you with any bugs or
difficulties you'll encounter :)

Moreover, if you are able to come up with some sort of scripts/recipe
for that, it will be the kind of documentation or tool that could
benefit big time the community! :)

Feel free to ask on the mailing list any concerns/questions/comments
you have.

Cheers!
David

Alawneh, Luay:
> Hi,
> 
> 
> 
> I am reading about LTTng and I would like to know if it can
> replace Log4c for logging user space events in the production
> environment. In addition to the method enter/exit events we also
> use Log4c to log many other events to indicate errors and other
> useful information.
> 
> 
> 
> To summarize, I would like to know if LTTng can be used in a
> production environment for logging different types of events. We
> are planning to replace Log4c with another logging framework to
> improve the performance by reducing the time spent in logging.
> 
> 
> 
> Thanks,
> 
> 
> 
> Luay Alawneh
> 
> Nuance Communications
> 
> 
> 
> 
> 
> This body part will be downloaded on demand.
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJQCXGLAAoJEELoaioR9I02u0YH/3MGDZJuzI6iTh4oVvXGeoAl
NkKjNOU6c/vrsPbys+sNnlczg8CyezT6nL8DOeLkhg/HBAS6JtJngULOwBaSiNV3
J4/SD6Y/Fa96oJnOR5pLUBVdLxrkdOhPvwEiUd2L1DIugrZKJ8pTxs28lX59ltq+
3s59b20pJw9hNEHeDRc7MUY5fEa6t0FNYONsnYqUEEyLeZwYydHz1offgTG2dAhC
FzUCsG1Xzq2TT5PkA/M9ZNu3YQlDJOOzPOY/9A+O9rI5993iAab/lZ/jaDAgZZ0j
4JpHzJGjaqF0XuY+3FpfoPC/DFqBBEjnzXzBTUQahmX7KjzqmdxQIwBiiiloMHg=
=Z97t
-----END PGP SIGNATURE-----

Re: LTTng

[Mathieu Desnoyers]

One small detail to keep in mind at this point: only the "discard" and
"overwrite" modes are implemented for handling buffer condition. If you
need to ensure that events are _never_ discarded, even with
high-throughput scenarios, we might need to implement a "blocking" mode,
which makes the application wait until there is space free in the buffer
before it continues. It would probably make more sense for logging.

Other than this detail, LTTng-UST would be a good fit for low-overhead
logging.

Thanks,

Mathieu

* Alawneh, Luay (Luay.Alawneh@nuance.com) wrote:
> Hi,
> 
> I am reading about LTTng and I would like to know if it can replace Log4c for logging user space events in the production environment. In addition to the method enter/exit events we also use Log4c to log many other events to indicate errors and other useful information.
> 
> To summarize, I would like to know if LTTng can be used in a production environment for logging different types of events. We are planning to replace Log4c with another logging framework to improve the performance by reducing the time spent in logging.
> 
> Thanks,
> 
> Luay Alawneh
> Nuance Communications
> 

> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev


-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com

Re: LTTng

[Yannick Brosseau]

[-- Attachment #1.1: Type: text/plain, Size: 2566 bytes --]


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

UST could work for your logging purpose, but one drag back is that
you'll need to have a session daemon running, which will spawn a
consumer. You will also need to manage the start/stop of the tracing.

Yannick

On 2012-07-20 10:56, David Goulet wrote:
> Hi Luay,
>
> I am not aware of any Log4c --> LTTng transition work done up to this
> day...
>
> One thing I can say is that user space tracing using LTTng will
> certainly improve speed and scalability performance.
>
> The event LOGLEVEL feature is pretty interesting for that where you
> can only enable a certain loglevel facility for a trace (exactly like
> the syslog facilities). You can refer to the lttng-ust(3) man page for
> more information.
>
> Moreover, you can also couple kernel traces to your user space traces
> which can give you way more information :).
>
> I guess that if now one replies to this thread describing their
> experience with log4c vs lttng, you guys will be the first to try a
> transition and we will be more than happy to help you with any bugs or
> difficulties you'll encounter :)
>
> Moreover, if you are able to come up with some sort of scripts/recipe
> for that, it will be the kind of documentation or tool that could
> benefit big time the community! :)
>
> Feel free to ask on the mailing list any concerns/questions/comments
> you have.
>
> Cheers!
> David
>
> Alawneh, Luay:
> > Hi,
>
>
>
> > I am reading about LTTng and I would like to know if it can
> > replace Log4c for logging user space events in the production
> > environment. In addition to the method enter/exit events we also
> > use Log4c to log many other events to indicate errors and other
> > useful information.
>
>
>
> > To summarize, I would like to know if LTTng can be used in a
> > production environment for logging different types of events. We
> > are planning to replace Log4c with another logging framework to
> > improve the performance by reducing the time spent in logging.
>
>
>
> > Thanks,
>
>
>
> > Luay Alawneh
>
> > Nuance Communications
>
>
>
>
>
> > This body part will be downloaded on demand.
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAlAJmNUACgkQFQrZ7GzHX2pCUQCfZDF5c6Bl+ums2gf3CfsBUxnB
pIYAn3KCWayyKnQDkqM/88663iWeFHFH
=z4nw
-----END PGP SIGNATURE-----


[-- Attachment #1.2: Type: text/html, Size: 4268 bytes --]

[-- Attachment #2: Type: text/plain, Size: 155 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: lttng

[Mathieu Desnoyers]

Hi Kubitz,

Sorry for the late reply, CCing lttng-dev, since this explanation about
robustness against crash can be useful to others,

* Kubitz Jörg (joerg.kubitz@hspg.de) wrote:
> You use per-cpu-datastructures only to prevent cache-line-bouncing and
> not for exclusive access?

For kernel tracing, we can disable preemption (and thus migration)
around accesses to per-cpu data structures. Therefore, it prevents both
cache line bouncing and ensures that only a single CPU accesses the
per-cpu data (although an IRQ/softirq could access the same per-cpu data
concurrently).

For user-space tracing, given that we cannot cheaply disable preemption,
we only use per-cpu data to prevent cache-line bouncing.

> Then you would need to atomically write the
> whole message.

This is correct. In both cases. We need to protect against interrupts in
the kernel, and against signals and other threads in user-space.

> That cannot be done if the message is bigger than the
> bus.

It's the space reservation that needs to be performed atomically, not
writing the message. Please see:

https://lttng.org/publications
M. Desnoyers, Low-Impact Operating System Tracing Ph.D. dissertation,
École Polytechnique de Montréal, 2009.

See the chapter describing the ring buffer.


> Or if you only reserve space atomically and the thread dies
> before it uses the reserved space the reader that waits for the insert
> is in deadlock.

This issue is not so relevant for kernel tracing, because we disable
preemption around use of the buffer, so only a kernel crash would cause
a thread to die between reserve and commit. But at that point, we have
bigger problems in the system than tracing.

For user-space tracing, this question is relevant, because we have to
deal with the fact that a user-space application can be killed at any
point.

There are now two tracing modes implemented in lttng-ust (user-space
tracer), each with their own advantages (+)/disadvantages(-):

1) per-pid buffer tracing,
  - each process in the system has its own per-cpu buffers,
    increased memory consumption, increased overhead for buffer
    allocation in workloads consisting of short-lived processes,
  + when a process dies, its associated wakeup pipe write-side is
    closed by the kernel, so the consumer knows it needs to read the
    last buffer. It can read it up to the last contiguous point in the
    last subbuffer where data was written (commit seq counter).

2) per-uid (shared) buffer tracing, (buffers shared across applications
   with same user ID)
  + less memory overhead, less runtime overhead for short-lived
    processes,
  - if a process dies between reserve and commit, it creates a whole in
    the subbuffer. This whole will be detected only when the producer
    will fill up the buffer and find the unbalanced reserved/commit
    counter the next time it reaches this sub-buffer. Until the buffer
    the unbalanced reserve/commit count is observed by a producer,
    the consumer is unable to read the buffer futher than the packet
    containing the "whole". When the producer rebalances this packet's
    reserve/commit count, it will be counted as a "discarded" packet,
    and we lose all data within the packet that contains the whole.

> So lttng is not robust?

It's actually a question of trade-offs: robustness (1) vs speed (2). We
provide configurations for both.

Thanks,

Mathieu

> 
> Thanks for your explanaition.
> 
> Jörg Kubitz

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: lttng

[Thibault, Daniel]

> Message: 1
> Date: Fri, 3 May 2013 10:42:22 -0400
>
> There are now two tracing modes implemented in lttng-ust (user-space tracer), each with their own advantages (+)/disadvantages(-):
>
> 1) per-pid buffer tracing,
> [...]
> 2) per-uid (shared) buffer tracing, (buffers shared across applications
>    with same user ID)
>   + less memory overhead, less runtime overhead for short-lived
>     processes,
>   - if a process dies between reserve and commit, it creates a whole in
>     the subbuffer. This whole will be detected only when the producer
>     will fill up the buffer and find the unbalanced reserved/commit
>     counter the next time it reaches this sub-buffer. Until the buffer
>     the unbalanced reserve/commit count is observed by a producer,
>     the consumer is unable to read the buffer futher than the packet
>     containing the "whole". When the producer rebalances this packet's
>     reserve/commit count, it will be counted as a "discarded" packet,
>     and we lose all data within the packet that contains the whole.

   whole ⇒ hole

   This changes the meaning completely.   :-)

Daniel U. Thibault
Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
2459 route de la Bravoure
Québec QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada | Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

LTTNG

[David Zafman]

On Ubuntu 12.04.1 LTS after doing an install-deps.sh and the new 
do_autogen.sh without -L, I get a config error with this in the config.log:

configure:22637: checking if lttng-gen-tp is sane
configure:22647: result: no
configure:22681: checking lttng/tracepoint.h usability
configure:22681: gcc -c  -g -Wextra -Wno-missing-field-initializers 
-Wno-missing-declarations -Wno-unused-parameter  conftest.c >&5
configure:22681: $? = 0
configure:22681: result: yes
configure:22681: checking lttng/tracepoint.h presence
configure:22681: gcc -E  conftest.c
configure:22681: $? = 0
configure:22681: result: yes
configure:22681: checking for lttng/tracepoint.h
configure:22681: result: yes
configure:22692: checking for lttng-gen-tp
configure:22708: found /usr/bin/lttng-gen-tp
configure:22719: result: yes
configure:22737: error: in `/home/dzafman/ceph2':
configure:22739: error: lttng-gen-tp does not behave properly

David Zafman
Senior Developer
http://www.redhat.com

Re: LTTNG

[David Zafman]

I did the following to get the latest stable release of LTTNG because 
the default Ubuntu packages weren't new enough.

Now do_autogen.sh works for me per Sage's change to make --with-lttng 
the default for developers.

sudo apt-add-repository ppa:lttng/ppa
sudo apt-get update
sudo apt-get install lttng-tools
sudo apt-get install liblttng-ust-dev

I skipped the following because I assume it is for kernel modules.
sudo apt-get install lttng-modules-dkms

David

On 2/3/15 11:36 AM, David Zafman wrote:
>
> On Ubuntu 12.04.1 LTS after doing an install-deps.sh and the new 
> do_autogen.sh without -L, I get a config error with this in the 
> config.log:
>
> configure:22637: checking if lttng-gen-tp is sane
> configure:22647: result: no
> configure:22681: checking lttng/tracepoint.h usability
> configure:22681: gcc -c  -g -Wextra -Wno-missing-field-initializers 
> -Wno-missing-declarations -Wno-unused-parameter  conftest.c >&5
> configure:22681: $? = 0
> configure:22681: result: yes
> configure:22681: checking lttng/tracepoint.h presence
> configure:22681: gcc -E  conftest.c
> configure:22681: $? = 0
> configure:22681: result: yes
> configure:22681: checking for lttng/tracepoint.h
> configure:22681: result: yes
> configure:22692: checking for lttng-gen-tp
> configure:22708: found /usr/bin/lttng-gen-tp
> configure:22719: result: yes
> configure:22737: error: in `/home/dzafman/ceph2':
> configure:22739: error: lttng-gen-tp does not behave properly
>
> David Zafman
> Senior Developer
> http://www.redhat.com
> -- 
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html