linux-perf-users.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [bug] nsinfo__mountns_enter failure can cause perf to operate on wrong file
@ 2023-10-27 15:58 Petr Špaček
  2023-11-06  4:32 ` Namhyung Kim
  0 siblings, 1 reply; 6+ messages in thread
From: Petr Špaček @ 2023-10-27 15:58 UTC (permalink / raw)
  To: linux-perf-users

Hello,

I've noticed that sometimes "perf record" confuses paths between two 
different namespaces and inadvertently operates on a wrong file (on the 
same path - but wrong mount namespace).

Version: 6.5, commit 750b95887e567848ac2c851dae47922cac6db946 from 
Linus's tree

Reproducer using Podman container:
All commands executed as non-privileged user, except the sysctl tweak.

$ sudo sysctl -w 'kernel.perf_event_paranoid = -1'

# binary in the container needs to have the same path as on host
# but different content
$ podman run -ti docker.io/library/ubuntu:23.10 /usr/bin/yes > /dev/null

# leave this running and get PID of the process inside the container
$ docker inspect $(docker ps --latest -q) | jq '.[0].State.Pid'

# ! record as an unprivileged user !
# record and use debuginfod from official Ubuntu servers
$ perf record --debuginfod='https://debuginfod.ubuntu.com/' --all-user 
-F 99 --pid 637217

# check if we got symbols ... we should have because debuginfod has been 
configured
$ perf script  # notice "unknown" all over the place

# check build-ids in the recording
$ perf buildid-list
f53cbc885777b8cfc9e54a8015318a71e6845bc3 /usr/bin/yes

# check build-id in the container
$ docker exec -ti $(docker ps --latest -q) bash -c 'apt update; apt 
install -yyy file; file `which yes`'
... BuildID[sha1]=7c895824831420bd30a372431d2b241bb6ff5554 ...


Values in perf output and in the containerized binary do not match. Huh? 
Where the "perf" value came from, anyway?

Turns out the buildid-list shows value from the _host_:
$ file /usr/bin/yes
... BuildID[sha1]=f53cbc885777b8cfc9e54a8015318a71e6845bc3 ...


My guess is that this can happen when the user executing "perf record" 
does not have privileges to nsenter() the target namespace, but this 
failure is not checked.

I could not follow "perf record" code, but "perf buildid-cache --add" 
calls nsinfo__mountns_enter() which has void return type, and hilarity 
ensues if that call fails.

It works as expected if I add "sudo" in front of each "perf" command.

I'm happy to assist with debugging and testing.

-- 
Petr Špaček
Internet Systems Consortium

^ permalink raw reply	[flat|nested] 6+ messages in thread
* [bug] nsinfo__mountns_enter failure can cause perf to operate on wrong file
@ 2023-10-30 16:42 Petr Špaček
  2023-10-30 19:40 ` Petr Špaček
  0 siblings, 1 reply; 6+ messages in thread
From: Petr Špaček @ 2023-10-30 16:42 UTC (permalink / raw)
  To: linux-perf-users

Hello,

I've noticed that sometimes "perf record" confuses paths between two 
different namespaces and inadvertently operates on a wrong file (on the 
same path - but wrong mount namespace).

Version: 6.5, commit 750b95887e567848ac2c851dae47922cac6db946 from 
Linus's tree

Reproducer using Podman container:
All commands executed as non-privileged user, except the sysctl tweak.

$ sudo sysctl -w 'kernel.perf_event_paranoid = -1'

# binary in the container needs to have the same path as on host
# but different content
$ podman run -ti docker.io/library/ubuntu:23.10 /usr/bin/yes > /dev/null

# leave this running and get PID of the process inside the container
$ docker inspect $(docker ps --latest -q) | jq '.[0].State.Pid'

# ! record as an unprivileged user !
# record and use debuginfod from official Ubuntu servers
$ perf record --debuginfod='https://debuginfod.ubuntu.com/' --all-user 
-F 99 --pid 637217

# check if we got symbols ... we should have because debuginfod has been 
configured
$ perf script  # notice "unknown" all over the place

# check build-ids in the recording
$ perf buildid-list
f53cbc885777b8cfc9e54a8015318a71e6845bc3 /usr/bin/yes

# check build-id in the container
$ docker exec -ti $(docker ps --latest -q) bash -c 'apt update; apt 
install -yyy file; file `which yes`'
... BuildID[sha1]=7c895824831420bd30a372431d2b241bb6ff5554 ...


Values in perf output and in the containerized binary do not match. Huh? 
Where the "perf" value came from, anyway?

Turns out the buildid-list shows value from the _host_:
$ file /usr/bin/yes
... BuildID[sha1]=f53cbc885777b8cfc9e54a8015318a71e6845bc3 ...


My guess is that this can happen when the user executing "perf record" 
does not have privileges to nsenter() the target namespace, but this 
failure is not checked.

I could not follow "perf record" code, but "perf buildid-cache --add" 
calls nsinfo__mountns_enter() which has void return type, and hilarity 
ensues if that call fails.

It works as expected if I add "sudo" in front of each "perf" command.

I'm happy to assist with debugging and testing.

-- 
Petr Špaček
Internet Systems Consortium

^ permalink raw reply	[flat|nested] 6+ messages in thread
* [bug] nsinfo__mountns_enter failure can cause perf to operate on wrong file
@ 2023-10-27 15:27 Petr Špaček
  0 siblings, 0 replies; 6+ messages in thread
From: Petr Špaček @ 2023-10-27 15:27 UTC (permalink / raw)
  To: linux-perf-users

Hello,

I've noticed that sometimes "perf record" confuses paths between two 
different namespaces and inadvertently operates on a wrong file (on the 
same path - but wrong mount namespace).

Version: 6.5, commit 750b95887e567848ac2c851dae47922cac6db946 from 
Linus's tree

Reproducer using Podman container:
All commands executed as non-privileged user, except the sysctl tweak.

$ sudo sysctl -w 'kernel.perf_event_paranoid = -1'

# binary in the container needs to have the same path as on host
# but different content
$ podman run -ti docker.io/library/ubuntu:23.10 /usr/bin/yes > /dev/null

# leave this running and get PID of the process inside the container
$ docker inspect $(docker ps --latest -q) | jq '.[0].State.Pid'

# ! record as an unprivileged user !
# record and use debuginfod from official Ubuntu servers
$ perf record --debuginfod='https://debuginfod.ubuntu.com/' --all-user 
-F 99 --pid 637217

# check if we got symbols ... we should have because debuginfod has been 
configured
$ perf script  # notice "unknown" all over the place

# check build-ids in the recording
$ perf buildid-list
f53cbc885777b8cfc9e54a8015318a71e6845bc3 /usr/bin/yes

# check build-id in the container
$ docker exec -ti $(docker ps --latest -q) bash -c 'apt update; apt 
install -yyy file; file `which yes`'
... BuildID[sha1]=7c895824831420bd30a372431d2b241bb6ff5554 ...


Values in perf output and in the containerized binary do not match. Huh? 
Where the "perf" value came from, anyway?

Turns out the buildid-list shows value from the _host_:
$ file /usr/bin/yes
... BuildID[sha1]=f53cbc885777b8cfc9e54a8015318a71e6845bc3 ...


My guess is that this can happen when the user executing "perf record" 
does not have privileges to nsenter() the target namespace, but this 
failure is not checked.

I could not follow "perf record" code, but "perf buildid-cache --add" 
calls nsinfo__mountns_enter() which has void return type, and hilarity 
ensues if that call fails.

I'm happy to assist with debugging and testing.

-- 
Petr Špaček
Internet Systems Consortium

^ permalink raw reply	[flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-11-20  9:28 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-10-27 15:58 [bug] nsinfo__mountns_enter failure can cause perf to operate on wrong file Petr Špaček
2023-11-06  4:32 ` Namhyung Kim
2023-11-20  9:28   ` Petr Špaček
  -- strict thread matches above, loose matches on Subject: below --
2023-10-30 16:42 Petr Špaček
2023-10-30 19:40 ` Petr Špaček
2023-10-27 15:27 Petr Špaček

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).