* [lttng-dev] Shared memory circular buffer and K8s pods @ 2022-10-18 13:00 Lemay, Paul via lttng-dev 2022-10-18 18:34 ` Jérémie Galarneau via lttng-dev 0 siblings, 1 reply; 3+ messages in thread From: Lemay, Paul via lttng-dev @ 2022-10-18 13:00 UTC (permalink / raw) To: lttng-dev@lists.lttng.org [-- Attachment #1.1: Type: text/plain, Size: 622 bytes --] Hello There, We are looking at LTTng for a development here at Dell. As I understand the LTTng, there are shared memories of circular buffer traces created between an application and the consumer daemon. After looking at some presentation that you did, I see that you have started working on containerization and pods. So our applications run in pods, and am not sure how to organize the pods knowing about those shared memories? Can you help us better understand how our code should be organized in the context of Kubernetes? Any documentation that we can access? Best Regards Internal Use - Confidential [-- Attachment #1.2: Type: text/html, Size: 2590 bytes --] [-- Attachment #2: Type: text/plain, Size: 156 bytes --] _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [lttng-dev] Shared memory circular buffer and K8s pods 2022-10-18 13:00 [lttng-dev] Shared memory circular buffer and K8s pods Lemay, Paul via lttng-dev @ 2022-10-18 18:34 ` Jérémie Galarneau via lttng-dev 2022-10-18 18:40 ` Lemay, Paul via lttng-dev 0 siblings, 1 reply; 3+ messages in thread From: Jérémie Galarneau via lttng-dev @ 2022-10-18 18:34 UTC (permalink / raw) To: Lemay, Paul; +Cc: lttng-dev@lists.lttng.org Hi Paul, We don’t have much documentation about deploying in Kubernetes; we’re working on it. It’s hard to provide very specific guidance without knowing your setup and constraints. Since you mention applications, I’m assuming you are mostly interested in user space tracing. There are two main approaches for that. First, you can deploy the full LTTng stack in every container: the daemons of LTTng-tools, and LTTng-UST + your app. That’s probably the simplest way to get going, but it goes against K8s’ model of having one process per container, which can become annoying. The other approach, which I would recommend you look into, is deploying an LTTng sidecar per pod that runs the session daemon, and installing LTTng-UST (the tracer libraries) along your instrumented applications in the other containers of the pod. Michael Jeanson has a demo showing how he deployed LTTng in K8s using that approach. Hopefully, that can help you get started: https://github.com/mjeanson/gcloud-lttng There are a couple things to note in this configuration: - A “memory” volume, devshm, is shared between containers. It has to be large enough to accommodate your buffer configuration, and - the containers share the same pid namespace across the pod (https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/). Then that leaves the question of where you want to store traces. There’s a ton of options there. By default, Michael's demo writes them to the lttng-home volume. You may want to look into network streaming to send traces over the network to a relay daemon (possibly running in another pod): https://lttng.org/docs/v2.13/#doc-lttng-relayd Otherwise, you may also be interested in session rotations to manage traces pretty much how you would handle logs: https://lttng.org/blog/2019/10/15/lttng-session-rotation/ https://lttng.org/docs/v2.13/#doc-session-rotation Let me know if you have more questions, Jérémie On Tue, Oct 18, 2022 at 01:00:47PM +0000, Lemay, Paul via lttng-dev wrote: > Hello There, > > We are looking at LTTng for a development here at Dell. As I understand the LTTng, there are shared memories of circular buffer traces created between an application and the consumer daemon. After looking at some presentation that you did, I see that you have started working on containerization and pods. > > So our applications run in pods, and am not sure how to organize the pods knowing about those shared memories? Can you help us better understand how our code should be organized in the context of Kubernetes? Any documentation that we can access? > > Best Regards > > > Internal Use - Confidential > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [lttng-dev] Shared memory circular buffer and K8s pods 2022-10-18 18:34 ` Jérémie Galarneau via lttng-dev @ 2022-10-18 18:40 ` Lemay, Paul via lttng-dev 0 siblings, 0 replies; 3+ messages in thread From: Lemay, Paul via lttng-dev @ 2022-10-18 18:40 UTC (permalink / raw) To: Jérémie Galarneau; +Cc: lttng-dev@lists.lttng.org Merci Jérémie! Pour te mettre en contexte, nous somme L'organisation télécom 5G de Dell. Je regarde LTTng présentement et je suis intéressé par la fonctone que vous décrivez comme étant "Flight Recorder". Je vais lire ces document et merci encore pour la réponse rapide! Internal Use - Confidential -----Original Message----- From: Jérémie Galarneau <jeremie.galarneau@efficios.com> Sent: Tuesday, October 18, 2022 2:34 PM To: Lemay, Paul Cc: lttng-dev@lists.lttng.org Subject: Re: [lttng-dev] Shared memory circular buffer and K8s pods [EXTERNAL EMAIL] Hi Paul, We don’t have much documentation about deploying in Kubernetes; we’re working on it. It’s hard to provide very specific guidance without knowing your setup and constraints. Since you mention applications, I’m assuming you are mostly interested in user space tracing. There are two main approaches for that. First, you can deploy the full LTTng stack in every container: the daemons of LTTng-tools, and LTTng-UST + your app. That’s probably the simplest way to get going, but it goes against K8s’ model of having one process per container, which can become annoying. The other approach, which I would recommend you look into, is deploying an LTTng sidecar per pod that runs the session daemon, and installing LTTng-UST (the tracer libraries) along your instrumented applications in the other containers of the pod. Michael Jeanson has a demo showing how he deployed LTTng in K8s using that approach. Hopefully, that can help you get started: https://urldefense.com/v3/__https://github.com/mjeanson/gcloud-lttng__;!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye8lWbYy9w$ [github[.]com] There are a couple things to note in this configuration: - A “memory” volume, devshm, is shared between containers. It has to be large enough to accommodate your buffer configuration, and - the containers share the same pid namespace across the pod (https://urldefense.com/v3/__https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/__;!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye-CE3jM7Q$ [kubernetes[.]io]). Then that leaves the question of where you want to store traces. There’s a ton of options there. By default, Michael's demo writes them to the lttng-home volume. You may want to look into network streaming to send traces over the network to a relay daemon (possibly running in another pod): https://urldefense.com/v3/__https://lttng.org/docs/v2.13/*doc-lttng-relayd__;Iw!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye8DDrqZHg$ [lttng[.]org] Otherwise, you may also be interested in session rotations to manage traces pretty much how you would handle logs: https://urldefense.com/v3/__https://lttng.org/blog/2019/10/15/lttng-session-rotation/__;!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye8jbW2hug$ [lttng[.]org] https://urldefense.com/v3/__https://lttng.org/docs/v2.13/*doc-session-rotation__;Iw!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye_2vJ9Z8g$ [lttng[.]org] Let me know if you have more questions, Jérémie On Tue, Oct 18, 2022 at 01:00:47PM +0000, Lemay, Paul via lttng-dev wrote: > Hello There, > > We are looking at LTTng for a development here at Dell. As I understand the LTTng, there are shared memories of circular buffer traces created between an application and the consumer daemon. After looking at some presentation that you did, I see that you have started working on containerization and pods. > > So our applications run in pods, and am not sure how to organize the pods knowing about those shared memories? Can you help us better understand how our code should be organized in the context of Kubernetes? Any documentation that we can access? > > Best Regards > > > Internal Use - Confidential > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://urldefense.com/v3/__https://lists.lttng.org/cgi-bin/mailman/li > stinfo/lttng-dev__;!!LpKI!hx1NeKvX04hHapBFJ4ZnTXZG3jluHVfm8fh_foCE_MH9 > KSgkf1VzKPXdopiYPQ3xt_G1B1atLn4jQ4IRye8k-nnC1A$ [lists[.]lttng[.]org] _______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2022-10-18 18:42 UTC | newest] Thread overview: 3+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2022-10-18 13:00 [lttng-dev] Shared memory circular buffer and K8s pods Lemay, Paul via lttng-dev 2022-10-18 18:34 ` Jérémie Galarneau via lttng-dev 2022-10-18 18:40 ` Lemay, Paul via lttng-dev
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.