[lttng-dev] Shared memory circular buffer and K8s pods

[lttng-dev] Shared memory circular buffer and K8s pods

[Lemay, Paul via lttng-dev]

[-- Attachment #1.1: Type: text/plain, Size: 622 bytes --]

Hello There,

We are looking at LTTng for a development here at Dell. As I understand the LTTng, there are shared memories of circular buffer traces created between  an application and the consumer daemon. After looking at some presentation that you did, I see that you have started working on containerization and pods.

So our applications run in pods, and am not sure how to organize the pods knowing about those shared memories? Can you help us better understand how our code should be organized in the context of Kubernetes? Any documentation that we can access?

Best Regards


Internal Use - Confidential

[-- Attachment #1.2: Type: text/html, Size: 2590 bytes --]

[-- Attachment #2: Type: text/plain, Size: 156 bytes --]

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Shared memory circular buffer and K8s pods

[Jérémie Galarneau via lttng-dev]

Hi Paul, 

We don’t have much documentation about deploying in Kubernetes; we’re working on
it.

It’s hard to provide very specific guidance without knowing your setup and
constraints. Since you mention applications, I’m assuming you are mostly
interested in user space tracing. There are two main approaches for that.

First, you can deploy the full LTTng stack in every container: the daemons of
LTTng-tools, and LTTng-UST + your app. That’s probably the simplest way to get
going, but it goes against K8s’ model of having one process per container, which
can become annoying.

The other approach, which I would recommend you look into, is deploying an LTTng
sidecar per pod that runs the session daemon, and installing LTTng-UST (the
tracer libraries) along your instrumented applications in the other containers
of the pod.

Michael Jeanson has a demo showing how he deployed LTTng in K8s using that
approach. Hopefully, that can help you get started:

https://github.com/mjeanson/gcloud-lttng

There are a couple things to note in this configuration:
  - A “memory” volume, devshm, is shared between containers. It has to be
    large enough to accommodate your buffer configuration, and
  - the containers share the same pid namespace across the pod
    (https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/).

Then that leaves the question of where you want to store traces. There’s a ton
of options there. By default, Michael's demo writes them to the lttng-home
volume. You may want to look into network streaming to send traces over the
network to a relay daemon (possibly running in another pod):

https://lttng.org/docs/v2.13/#doc-lttng-relayd

Otherwise, you may also be interested in session rotations to manage traces
pretty much how you would handle logs:

https://lttng.org/blog/2019/10/15/lttng-session-rotation/

https://lttng.org/docs/v2.13/#doc-session-rotation

Let me know if you have more questions,
Jérémie

On Tue, Oct 18, 2022 at 01:00:47PM +0000, Lemay, Paul via lttng-dev wrote:
> Hello There,
> 
> We are looking at LTTng for a development here at Dell. As I understand the LTTng, there are shared memories of circular buffer traces created between  an application and the consumer daemon. After looking at some presentation that you did, I see that you have started working on containerization and pods.
> 
> So our applications run in pods, and am not sure how to organize the pods knowing about those shared memories? Can you help us better understand how our code should be organized in the context of Kubernetes? Any documentation that we can access?
> 
> Best Regards
> 
> 
> Internal Use - Confidential

> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

Re: [lttng-dev] Shared memory circular buffer and K8s pods

[Lemay, Paul via lttng-dev]

Merci Jérémie!
Pour te mettre en contexte, nous somme L'organisation télécom 5G de Dell. Je regarde LTTng présentement et je suis intéressé par la fonctone que vous décrivez comme étant "Flight Recorder". 

Je vais lire ces document et merci encore pour la réponse rapide!


Internal Use - Confidential

-----Original Message-----
From: Jérémie Galarneau <jeremie.galarneau@efficios.com> 
Sent: Tuesday, October 18, 2022 2:34 PM
To: Lemay, Paul
Cc: lttng-dev@lists.lttng.org
Subject: Re: [lttng-dev] Shared memory circular buffer and K8s pods


[EXTERNAL EMAIL] 

Hi Paul, 

We don’t have much documentation about deploying in Kubernetes; we’re working on it.

It’s hard to provide very specific guidance without knowing your setup and constraints. Since you mention applications, I’m assuming you are mostly interested in user space tracing. There are two main approaches for that.

First, you can deploy the full LTTng stack in every container: the daemons of LTTng-tools, and LTTng-UST + your app. That’s probably the simplest way to get going, but it goes against K8s’ model of having one process per container, which can become annoying.

The other approach, which I would recommend you look into, is deploying an LTTng sidecar per pod that runs the session daemon, and installing LTTng-UST (the tracer libraries) along your instrumented applications in the other containers of the pod.

Michael Jeanson has a demo showing how he deployed LTTng in K8s using that approach. Hopefully, that can help you get started:

https://urldefense.com/v3/__https://github.com/mjeanson/gcloud-lttng__;!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye8lWbYy9w$  [github[.]com]

There are a couple things to note in this configuration:
  - A “memory” volume, devshm, is shared between containers. It has to be
    large enough to accommodate your buffer configuration, and
  - the containers share the same pid namespace across the pod
    (https://urldefense.com/v3/__https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/__;!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye-CE3jM7Q$  [kubernetes[.]io]).

Then that leaves the question of where you want to store traces. There’s a ton of options there. By default, Michael's demo writes them to the lttng-home volume. You may want to look into network streaming to send traces over the network to a relay daemon (possibly running in another pod):

https://urldefense.com/v3/__https://lttng.org/docs/v2.13/*doc-lttng-relayd__;Iw!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye8DDrqZHg$  [lttng[.]org]

Otherwise, you may also be interested in session rotations to manage traces pretty much how you would handle logs:

https://urldefense.com/v3/__https://lttng.org/blog/2019/10/15/lttng-session-rotation/__;!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye8jbW2hug$  [lttng[.]org]

https://urldefense.com/v3/__https://lttng.org/docs/v2.13/*doc-session-rotation__;Iw!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye_2vJ9Z8g$  [lttng[.]org]

Let me know if you have more questions,
Jérémie

On Tue, Oct 18, 2022 at 01:00:47PM +0000, Lemay, Paul via lttng-dev wrote:
> Hello There,
> 
> We are looking at LTTng for a development here at Dell. As I understand the LTTng, there are shared memories of circular buffer traces created between  an application and the consumer daemon. After looking at some presentation that you did, I see that you have started working on containerization and pods.
> 
> So our applications run in pods, and am not sure how to organize the pods knowing about those shared memories? Can you help us better understand how our code should be organized in the context of Kubernetes? Any documentation that we can access?
> 
> Best Regards
> 
> 
> Internal Use - Confidential

> _______________________________________________
> lttng-dev mailing list
> lttng-dev@lists.lttng.org
> https://urldefense.com/v3/__https://lists.lttng.org/cgi-bin/mailman/li
> stinfo/lttng-dev__;!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9
> KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye8k-nnC1A$  [lists[.]lttng[.]org]
_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev