Kernel tracing from inside a Docker container

Kernel tracing from inside a Docker container

[Dirk Eibach via lttng-dev]

Hi,

sorry, Outlook is messing up my plaintext mails, so I am switching to
my gmail account.

> lttng-modules exposes its control ABI through the virtual file /proc/lttng.
>
> Does you container have access to that file ?

The container is running in privileged mode.
root@9348eb19e6f4:/# ls -l /proc/lttng
-rw------- 1 root root 0 Nov  6 16:05 /proc/lttng

> Note that if you give access to that file within the container, then the container
> can observe detailed information about kernel internals (including related to
> other containers).

This is a development-only system, so the security implications are acceptable.

I have observed one more thing. When I start the session daemon with
the -vvv option I get:
root@9348eb19e6f4:/# lttng list --kernel
DBG1 - 07:31:23.811796947 [Client management]: Wait for client
response (in thread_manage_clients() at client.c:2576)
DBG1 - 07:31:23.811867247 [Client management]: Receiving data from
client ... (in thread_manage_clients() at client.c:2604)
DBG1 - 07:31:23.811945247 [Client management]: Processing client
command 'LTTNG_LIST_TRACEPOINTS' (14) (in process_client_msg() at
client.c:1016)
Error: Failed to load kmod library resources
Warning: No kernel tracer available
DBG1 - 07:31:23.812076447 [Client management]: Missing llm header,
creating one. (in process_client_msg() at client.c:2381)
DBG1 - 07:31:23.812085547 [Client management]: Sending response (size:
24, retcode: Kernel tracer not available (33)) (in
thread_manage_clients() at client.c:2683)
DBG1 - 07:31:23.812156547 [Client management]: Accepting client
command ... (in thread_manage_clients() at client.c:2534)
Error: Unable to list kernel events: Kernel tracer not available

Regards
Dirk

Kernel tracing from inside a Docker container

[Dirk Eibach via lttng-dev]

Hi,

sorry, Outlook is messing up my plaintext mails, so I am switching to
my gmail account.

> lttng-modules exposes its control ABI through the virtual file /proc/lttng.
>
> Does you container have access to that file ?

The container is running in privileged mode.
root@9348eb19e6f4:/# ls -l /proc/lttng
-rw------- 1 root root 0 Nov  6 16:05 /proc/lttng

> Note that if you give access to that file within the container, then the container
> can observe detailed information about kernel internals (including related to
> other containers).

This is a development-only system, so the security implications are acceptable.

I have observed one more thing. When I start the session daemon with
the -vvv option I get:
root@9348eb19e6f4:/# lttng list --kernel
DBG1 - 07:31:23.811796947 [Client management]: Wait for client
response (in thread_manage_clients() at client.c:2576)
DBG1 - 07:31:23.811867247 [Client management]: Receiving data from
client ... (in thread_manage_clients() at client.c:2604)
DBG1 - 07:31:23.811945247 [Client management]: Processing client
command 'LTTNG_LIST_TRACEPOINTS' (14) (in process_client_msg() at
client.c:1016)
Error: Failed to load kmod library resources
Warning: No kernel tracer available
DBG1 - 07:31:23.812076447 [Client management]: Missing llm header,
creating one. (in process_client_msg() at client.c:2381)
DBG1 - 07:31:23.812085547 [Client management]: Sending response (size:
24, retcode: Kernel tracer not available (33)) (in
thread_manage_clients() at client.c:2683)
DBG1 - 07:31:23.812156547 [Client management]: Accepting client
command ... (in thread_manage_clients() at client.c:2534)
Error: Unable to list kernel events: Kernel tracer not available

Regards
Dirk

Re: Kernel tracing from inside a Docker container

[Mathieu Desnoyers via lttng-dev]

On 2024-11-07 03:03, Dirk Eibach via lttng-dev wrote:
> Hi,
> 
> sorry, Outlook is messing up my plaintext mails, so I am switching to
> my gmail account.
> 
>> lttng-modules exposes its control ABI through the virtual file /proc/lttng.
>>
>> Does you container have access to that file ?
> 
> The container is running in privileged mode.
> root@9348eb19e6f4:/# ls -l /proc/lttng
> -rw------- 1 root root 0 Nov  6 16:05 /proc/lttng
> 
>> Note that if you give access to that file within the container, then the container
>> can observe detailed information about kernel internals (including related to
>> other containers).
> 
> This is a development-only system, so the security implications are acceptable.
> 
> I have observed one more thing. When I start the session daemon with
> the -vvv option I get:
> root@9348eb19e6f4:/# lttng list --kernel
> DBG1 - 07:31:23.811796947 [Client management]: Wait for client
> response (in thread_manage_clients() at client.c:2576)
> DBG1 - 07:31:23.811867247 [Client management]: Receiving data from
> client ... (in thread_manage_clients() at client.c:2604)
> DBG1 - 07:31:23.811945247 [Client management]: Processing client
> command 'LTTNG_LIST_TRACEPOINTS' (14) (in process_client_msg() at
> client.c:1016)
> Error: Failed to load kmod library resources
> Warning: No kernel tracer available

I suspect that you have two choices here:

1) You modprobe all LTTng modules from your top level namespace (host
    system) before starting lttng-sessiond within the container, or

2) You ensure that the container has the ability to load kernel
    modules, and you make sure the lttng-modules .ko are available
    in the container's under /lib/modules/$(uname -r)/ directory.
    (and run depmod -a), then launch lttng-sessiond as root.

Thanks,

Mathieu

> DBG1 - 07:31:23.812076447 [Client management]: Missing llm header,
> creating one. (in process_client_msg() at client.c:2381)
> DBG1 - 07:31:23.812085547 [Client management]: Sending response (size:
> 24, retcode: Kernel tracer not available (33)) (in
> thread_manage_clients() at client.c:2683)
> DBG1 - 07:31:23.812156547 [Client management]: Accepting client
> command ... (in thread_manage_clients() at client.c:2534)
> Error: Unable to list kernel events: Kernel tracer not available
> 
> Regards
> Dirk

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

Re: Kernel tracing from inside a Docker container

[Dirk Eibach via lttng-dev]

Hi Mathieu,

> 1) You modprobe all LTTng modules from your top level namespace (host
>     system) before starting lttng-sessiond within the container, or

since I already did a lttng session on the host/top level namespacer
the modules are already loaded:

root@9348eb19e6f4:/# lsmod
Module                  Size  Used by
lttng_probe_x86_exceptions    16384  0
lttng_probe_x86_irq_vectors    16384  0
lttng_probe_workqueue    20480  0
lttng_probe_vmscan     32768  0
lttng_probe_udp        16384  0
lttng_probe_timer      28672  0
lttng_probe_sunrpc     20480  0
lttng_probe_statedump    45056  0
lttng_probe_sock       16384  0
lttng_probe_skb        16384  0
lttng_probe_signal     16384  0
lttng_probe_scsi       20480  0
lttng_probe_sched      40960  0
lttng_probe_rcu        16384  0
lttng_probe_printk     16384  0
lttng_probe_power      20480  0
lttng_probe_net        28672  0
lttng_probe_napi       16384  0
lttng_probe_module     20480  0
lttng_probe_kvm        32768  0
lttng_probe_kmem       28672  0
lttng_probe_jbd2       28672  0
lttng_probe_irq        16384  0
lttng_probe_i2c        20480  0
lttng_probe_ext4      155648  0
lttng_probe_compaction    16384  0
lttng_probe_btrfs      65536  0
lttng_probe_block      36864  0
lttng_counter_client_percpu_32_modular    16384  0
lttng_counter_client_percpu_64_modular    16384  1
lttng_counter          16384  2
lttng_counter_client_percpu_64_modular,lttng_counter_client_percpu_32_modular
lttng_ring_buffer_event_notifier_client    16384  2
lttng_ring_buffer_metadata_mmap_client    16384  0
lttng_ring_buffer_client_mmap_overwrite    20480  0
lttng_ring_buffer_client_mmap_discard    20480  0
lttng_ring_buffer_metadata_client    16384  0
lttng_ring_buffer_client_overwrite    20480  0
lttng_ring_buffer_client_discard    20480  0
lttng_tracer         2666496  40
lttng_probe_udp,lttng_probe_scsi,lttng_probe_sched,lttng_probe_compaction,lttng_probe_net,lttng_probe_vmscan,lttng_probe_x86_irq_vectors,lttng_probe_power,lttng_probe_rcu,lttng_probe_module,lttng_ring_buffer_client_mmap_overwrite,lttng_probe_statedump,lttng_ring_buffer_client_discard,lttng_probe_printk,lttng_probe_x86_exceptions,lttng_probe_ext4,lttng_probe_sock,lttng_counter_client_percpu_64_modular,lttng_probe_irq,lttng_ring_buffer_client_mmap_discard,lttng_probe_kvm,lttng_probe_timer,lttng_probe_btrfs,lttng_ring_buffer_event_notifier_client,lttng_counter_client_percpu_32_modular,lttng_probe_workqueue,lttng_probe_jbd2,lttng_probe_i2c,lttng_probe_signal,lttng_probe_skb,lttng_probe_block,lttng_probe_napi,lttng_ring_buffer_metadata_client,lttng_probe_kmem,lttng_ring_buffer_metadata_mmap_client,lttng_ring_buffer_client_overwrite,lttng_probe_sunrpc
lttng_statedump        49152  1 lttng_tracer
lttng_wrapper          16384  7
lttng_statedump,lttng_ring_buffer_client_mmap_overwrite,lttng_ring_buffer_client_discard,lttng_tracer,lttng_ring_buffer_client_mmap_discard,lttng_probe_kmem,lttng_ring_buffer_client_overwrite
lttng_clock            16384  5
lttng_ring_buffer_client_mmap_overwrite,lttng_ring_buffer_client_discard,lttng_tracer,lttng_ring_buffer_client_mmap_discard,lttng_ring_buffer_client_overwrite
lttng_lib_ring_buffer    61440  8
lttng_ring_buffer_client_mmap_overwrite,lttng_ring_buffer_client_discard,lttng_tracer,lttng_ring_buffer_client_mmap_discard,lttng_ring_buffer_event_notifier_client,lttng_ring_buffer_metadata_client,lttng_ring_buffer_metadata_mmap_client,lttng_ring_buffer_client_overwrite

Thanks
Dirk

On Thu, Nov 7, 2024 at 3:29 PM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> On 2024-11-07 03:03, Dirk Eibach via lttng-dev wrote:
> > Hi,
> >
> > sorry, Outlook is messing up my plaintext mails, so I am switching to
> > my gmail account.
> >
> >> lttng-modules exposes its control ABI through the virtual file /proc/lttng.
> >>
> >> Does you container have access to that file ?
> >
> > The container is running in privileged mode.
> > root@9348eb19e6f4:/# ls -l /proc/lttng
> > -rw------- 1 root root 0 Nov  6 16:05 /proc/lttng
> >
> >> Note that if you give access to that file within the container, then the container
> >> can observe detailed information about kernel internals (including related to
> >> other containers).
> >
> > This is a development-only system, so the security implications are acceptable.
> >
> > I have observed one more thing. When I start the session daemon with
> > the -vvv option I get:
> > root@9348eb19e6f4:/# lttng list --kernel
> > DBG1 - 07:31:23.811796947 [Client management]: Wait for client
> > response (in thread_manage_clients() at client.c:2576)
> > DBG1 - 07:31:23.811867247 [Client management]: Receiving data from
> > client ... (in thread_manage_clients() at client.c:2604)
> > DBG1 - 07:31:23.811945247 [Client management]: Processing client
> > command 'LTTNG_LIST_TRACEPOINTS' (14) (in process_client_msg() at
> > client.c:1016)
> > Error: Failed to load kmod library resources
> > Warning: No kernel tracer available
>
> I suspect that you have two choices here:
>
> 1) You modprobe all LTTng modules from your top level namespace (host
>     system) before starting lttng-sessiond within the container, or
>
> 2) You ensure that the container has the ability to load kernel
>     modules, and you make sure the lttng-modules .ko are available
>     in the container's under /lib/modules/$(uname -r)/ directory.
>     (and run depmod -a), then launch lttng-sessiond as root.
>
> Thanks,
>
> Mathieu
>
> > DBG1 - 07:31:23.812076447 [Client management]: Missing llm header,
> > creating one. (in process_client_msg() at client.c:2381)
> > DBG1 - 07:31:23.812085547 [Client management]: Sending response (size:
> > 24, retcode: Kernel tracer not available (33)) (in
> > thread_manage_clients() at client.c:2683)
> > DBG1 - 07:31:23.812156547 [Client management]: Accepting client
> > command ... (in thread_manage_clients() at client.c:2534)
> > Error: Unable to list kernel events: Kernel tracer not available
> >
> > Regards
> > Dirk
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> https://www.efficios.com
>

Re: Kernel tracing from inside a Docker container

[Dirk Eibach via lttng-dev]

Hi,

I guess I found it! The kernel module files have to be present under
/lib/modules inside the container, even when they are already loaded
on the host side.
After putting them there, everything works fine and I finally can
record kernel events.

Thank you so much for your support.

Regards
Dirk

On Thu, Nov 7, 2024 at 3:58 PM Dirk Eibach <dirk.eibach@googlemail.com> wrote:
>
> Hi Mathieu,
>
> > 1) You modprobe all LTTng modules from your top level namespace (host
> >     system) before starting lttng-sessiond within the container, or
>
> since I already did a lttng session on the host/top level namespacer
> the modules are already loaded:
>
> root@9348eb19e6f4:/# lsmod
> Module                  Size  Used by
> lttng_probe_x86_exceptions    16384  0
> lttng_probe_x86_irq_vectors    16384  0
> lttng_probe_workqueue    20480  0
> lttng_probe_vmscan     32768  0
> lttng_probe_udp        16384  0
> lttng_probe_timer      28672  0
> lttng_probe_sunrpc     20480  0
> lttng_probe_statedump    45056  0
> lttng_probe_sock       16384  0
> lttng_probe_skb        16384  0
> lttng_probe_signal     16384  0
> lttng_probe_scsi       20480  0
> lttng_probe_sched      40960  0
> lttng_probe_rcu        16384  0
> lttng_probe_printk     16384  0
> lttng_probe_power      20480  0
> lttng_probe_net        28672  0
> lttng_probe_napi       16384  0
> lttng_probe_module     20480  0
> lttng_probe_kvm        32768  0
> lttng_probe_kmem       28672  0
> lttng_probe_jbd2       28672  0
> lttng_probe_irq        16384  0
> lttng_probe_i2c        20480  0
> lttng_probe_ext4      155648  0
> lttng_probe_compaction    16384  0
> lttng_probe_btrfs      65536  0
> lttng_probe_block      36864  0
> lttng_counter_client_percpu_32_modular    16384  0
> lttng_counter_client_percpu_64_modular    16384  1
> lttng_counter          16384  2
> lttng_counter_client_percpu_64_modular,lttng_counter_client_percpu_32_modular
> lttng_ring_buffer_event_notifier_client    16384  2
> lttng_ring_buffer_metadata_mmap_client    16384  0
> lttng_ring_buffer_client_mmap_overwrite    20480  0
> lttng_ring_buffer_client_mmap_discard    20480  0
> lttng_ring_buffer_metadata_client    16384  0
> lttng_ring_buffer_client_overwrite    20480  0
> lttng_ring_buffer_client_discard    20480  0
> lttng_tracer         2666496  40
> lttng_probe_udp,lttng_probe_scsi,lttng_probe_sched,lttng_probe_compaction,lttng_probe_net,lttng_probe_vmscan,lttng_probe_x86_irq_vectors,lttng_probe_power,lttng_probe_rcu,lttng_probe_module,lttng_ring_buffer_client_mmap_overwrite,lttng_probe_statedump,lttng_ring_buffer_client_discard,lttng_probe_printk,lttng_probe_x86_exceptions,lttng_probe_ext4,lttng_probe_sock,lttng_counter_client_percpu_64_modular,lttng_probe_irq,lttng_ring_buffer_client_mmap_discard,lttng_probe_kvm,lttng_probe_timer,lttng_probe_btrfs,lttng_ring_buffer_event_notifier_client,lttng_counter_client_percpu_32_modular,lttng_probe_workqueue,lttng_probe_jbd2,lttng_probe_i2c,lttng_probe_signal,lttng_probe_skb,lttng_probe_block,lttng_probe_napi,lttng_ring_buffer_metadata_client,lttng_probe_kmem,lttng_ring_buffer_metadata_mmap_client,lttng_ring_buffer_client_overwrite,lttng_probe_sunrpc
> lttng_statedump        49152  1 lttng_tracer
> lttng_wrapper          16384  7
> lttng_statedump,lttng_ring_buffer_client_mmap_overwrite,lttng_ring_buffer_client_discard,lttng_tracer,lttng_ring_buffer_client_mmap_discard,lttng_probe_kmem,lttng_ring_buffer_client_overwrite
> lttng_clock            16384  5
> lttng_ring_buffer_client_mmap_overwrite,lttng_ring_buffer_client_discard,lttng_tracer,lttng_ring_buffer_client_mmap_discard,lttng_ring_buffer_client_overwrite
> lttng_lib_ring_buffer    61440  8
> lttng_ring_buffer_client_mmap_overwrite,lttng_ring_buffer_client_discard,lttng_tracer,lttng_ring_buffer_client_mmap_discard,lttng_ring_buffer_event_notifier_client,lttng_ring_buffer_metadata_client,lttng_ring_buffer_metadata_mmap_client,lttng_ring_buffer_client_overwrite
>
> Thanks
> Dirk
>
> On Thu, Nov 7, 2024 at 3:29 PM Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:
> >
> > On 2024-11-07 03:03, Dirk Eibach via lttng-dev wrote:
> > > Hi,
> > >
> > > sorry, Outlook is messing up my plaintext mails, so I am switching to
> > > my gmail account.
> > >
> > >> lttng-modules exposes its control ABI through the virtual file /proc/lttng.
> > >>
> > >> Does you container have access to that file ?
> > >
> > > The container is running in privileged mode.
> > > root@9348eb19e6f4:/# ls -l /proc/lttng
> > > -rw------- 1 root root 0 Nov  6 16:05 /proc/lttng
> > >
> > >> Note that if you give access to that file within the container, then the container
> > >> can observe detailed information about kernel internals (including related to
> > >> other containers).
> > >
> > > This is a development-only system, so the security implications are acceptable.
> > >
> > > I have observed one more thing. When I start the session daemon with
> > > the -vvv option I get:
> > > root@9348eb19e6f4:/# lttng list --kernel
> > > DBG1 - 07:31:23.811796947 [Client management]: Wait for client
> > > response (in thread_manage_clients() at client.c:2576)
> > > DBG1 - 07:31:23.811867247 [Client management]: Receiving data from
> > > client ... (in thread_manage_clients() at client.c:2604)
> > > DBG1 - 07:31:23.811945247 [Client management]: Processing client
> > > command 'LTTNG_LIST_TRACEPOINTS' (14) (in process_client_msg() at
> > > client.c:1016)
> > > Error: Failed to load kmod library resources
> > > Warning: No kernel tracer available
> >
> > I suspect that you have two choices here:
> >
> > 1) You modprobe all LTTng modules from your top level namespace (host
> >     system) before starting lttng-sessiond within the container, or
> >
> > 2) You ensure that the container has the ability to load kernel
> >     modules, and you make sure the lttng-modules .ko are available
> >     in the container's under /lib/modules/$(uname -r)/ directory.
> >     (and run depmod -a), then launch lttng-sessiond as root.
> >
> > Thanks,
> >
> > Mathieu
> >
> > > DBG1 - 07:31:23.812076447 [Client management]: Missing llm header,
> > > creating one. (in process_client_msg() at client.c:2381)
> > > DBG1 - 07:31:23.812085547 [Client management]: Sending response (size:
> > > 24, retcode: Kernel tracer not available (33)) (in
> > > thread_manage_clients() at client.c:2683)
> > > DBG1 - 07:31:23.812156547 [Client management]: Accepting client
> > > command ... (in thread_manage_clients() at client.c:2534)
> > > Error: Unable to list kernel events: Kernel tracer not available
> > >
> > > Regards
> > > Dirk
> >
> > --
> > Mathieu Desnoyers
> > EfficiOS Inc.
> > https://www.efficios.com
> >

Re: Kernel tracing from inside a Docker container

[Michael Jeanson via lttng-dev]

On 2024-11-07 11:00, Dirk Eibach via lttng-dev wrote:
> Hi,
> 
> I guess I found it! The kernel module files have to be present under
> /lib/modules inside the container, even when they are already loaded
> on the host side.
> After putting them there, everything works fine and I finally can
> record kernel events.
> 
> Thank you so much for your support.

I just looked at the code that loads the modules and we do fail early
if libkmod fails to initialize. I think we could just print a warning
there instead and still try to open '/proc/lttng' and then fail if it's
not present.

Re: Kernel tracing from inside a Docker container

[Mathieu Desnoyers via lttng-dev]

Hi,

lttng-modules exposes its control ABI through the
virtual file /proc/lttng.

Does you container have access to that file ?

Note that if you give access to that file within the
container, then the container can observe detailed
information about kernel internals (including related
to other containers).

Thanks,

Mathieu

On 2024-11-06 02:38, Eibach, Dirk via lttng-dev wrote:
> Hi,
> 
> I want to do kernel tracing from inside a Docker container. 
> Kernel-tracing works from the host and the modules are loaded.
> Userspace tracing works inside the container. But when I do "lttng list 
> --kernel" from the container I get "Error: Unable to list kernel events: 
> Kernel tracer not available". Running in the container in priviledged 
> mode makes no difference.
> 
> Mit freundlichen Grüßen / Best Regards
> Dirk Eibach
> _________________________________
> CARL CLOOS SCHWEISSTECHNIK GMBH
> Softwareentwickler / AU-EW
> Softwaredeveloper / AU-EW
> Hauptverwaltung/Headquarters: Carl-Cloos-Strasse 1
> Zentrallager/Central Warehouse: Carl-Cloos-Strasse 6
> 35708 Haiger
> Germany
> Tel:      +49 2773 85-888
> E-Mail: mailto:dirk.eibach@cloos.de <mailto:dirk.eibach@cloos.de>
> Internet: http://www.cloos.de/ <http://www.cloos.de/>
> 
> Sitz der Firma/Headquarters: 35708 Haiger, Germany
> Amtsgericht/Lower district court: Wetzlar HR B 3052
> UST.-ID-Nr./VAT ID No.: DE 111 79 45 11
> WEEE-Reg.-Nr./WEEE-Reg. No.: DE83919745
> Geschäftsführer/Management: Stephan Pittner (CEO)
> 
> <https://www.linkedin.com/company/carl-cloos-schwei%C3%9Ftechnik-gmbh> 
> <https://www.youtube.com/user/CloosWelding> 
> <https://www.facebook.com/CloosWelding> 
> <https://www.instagram.com/CloosWelding> 
> <https://www.xing.com/profile/carlcloosschweisstechnikgmbh>
> 
> CLOOS-Newsletter <https://www.cloos.de/de-de/#newsletter>
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com