* [bug] nsinfo__mountns_enter failure can cause perf to operate on wrong file
@ 2023-10-27 15:27 Petr Špaček
0 siblings, 0 replies; 6+ messages in thread
From: Petr Špaček @ 2023-10-27 15:27 UTC (permalink / raw)
To: linux-perf-users
Hello,
I've noticed that sometimes "perf record" confuses paths between two
different namespaces and inadvertently operates on a wrong file (on the
same path - but wrong mount namespace).
Version: 6.5, commit 750b95887e567848ac2c851dae47922cac6db946 from
Linus's tree
Reproducer using Podman container:
All commands executed as non-privileged user, except the sysctl tweak.
$ sudo sysctl -w 'kernel.perf_event_paranoid = -1'
# binary in the container needs to have the same path as on host
# but different content
$ podman run -ti docker.io/library/ubuntu:23.10 /usr/bin/yes > /dev/null
# leave this running and get PID of the process inside the container
$ docker inspect $(docker ps --latest -q) | jq '.[0].State.Pid'
# ! record as an unprivileged user !
# record and use debuginfod from official Ubuntu servers
$ perf record --debuginfod='https://debuginfod.ubuntu.com/' --all-user
-F 99 --pid 637217
# check if we got symbols ... we should have because debuginfod has been
configured
$ perf script # notice "unknown" all over the place
# check build-ids in the recording
$ perf buildid-list
f53cbc885777b8cfc9e54a8015318a71e6845bc3 /usr/bin/yes
# check build-id in the container
$ docker exec -ti $(docker ps --latest -q) bash -c 'apt update; apt
install -yyy file; file `which yes`'
... BuildID[sha1]=7c895824831420bd30a372431d2b241bb6ff5554 ...
Values in perf output and in the containerized binary do not match. Huh?
Where the "perf" value came from, anyway?
Turns out the buildid-list shows value from the _host_:
$ file /usr/bin/yes
... BuildID[sha1]=f53cbc885777b8cfc9e54a8015318a71e6845bc3 ...
My guess is that this can happen when the user executing "perf record"
does not have privileges to nsenter() the target namespace, but this
failure is not checked.
I could not follow "perf record" code, but "perf buildid-cache --add"
calls nsinfo__mountns_enter() which has void return type, and hilarity
ensues if that call fails.
I'm happy to assist with debugging and testing.
--
Petr Špaček
Internet Systems Consortium
^ permalink raw reply [flat|nested] 6+ messages in thread
* [bug] nsinfo__mountns_enter failure can cause perf to operate on wrong file
@ 2023-10-27 15:58 Petr Špaček
2023-11-06 4:32 ` Namhyung Kim
0 siblings, 1 reply; 6+ messages in thread
From: Petr Špaček @ 2023-10-27 15:58 UTC (permalink / raw)
To: linux-perf-users
Hello,
I've noticed that sometimes "perf record" confuses paths between two
different namespaces and inadvertently operates on a wrong file (on the
same path - but wrong mount namespace).
Version: 6.5, commit 750b95887e567848ac2c851dae47922cac6db946 from
Linus's tree
Reproducer using Podman container:
All commands executed as non-privileged user, except the sysctl tweak.
$ sudo sysctl -w 'kernel.perf_event_paranoid = -1'
# binary in the container needs to have the same path as on host
# but different content
$ podman run -ti docker.io/library/ubuntu:23.10 /usr/bin/yes > /dev/null
# leave this running and get PID of the process inside the container
$ docker inspect $(docker ps --latest -q) | jq '.[0].State.Pid'
# ! record as an unprivileged user !
# record and use debuginfod from official Ubuntu servers
$ perf record --debuginfod='https://debuginfod.ubuntu.com/' --all-user
-F 99 --pid 637217
# check if we got symbols ... we should have because debuginfod has been
configured
$ perf script # notice "unknown" all over the place
# check build-ids in the recording
$ perf buildid-list
f53cbc885777b8cfc9e54a8015318a71e6845bc3 /usr/bin/yes
# check build-id in the container
$ docker exec -ti $(docker ps --latest -q) bash -c 'apt update; apt
install -yyy file; file `which yes`'
... BuildID[sha1]=7c895824831420bd30a372431d2b241bb6ff5554 ...
Values in perf output and in the containerized binary do not match. Huh?
Where the "perf" value came from, anyway?
Turns out the buildid-list shows value from the _host_:
$ file /usr/bin/yes
... BuildID[sha1]=f53cbc885777b8cfc9e54a8015318a71e6845bc3 ...
My guess is that this can happen when the user executing "perf record"
does not have privileges to nsenter() the target namespace, but this
failure is not checked.
I could not follow "perf record" code, but "perf buildid-cache --add"
calls nsinfo__mountns_enter() which has void return type, and hilarity
ensues if that call fails.
It works as expected if I add "sudo" in front of each "perf" command.
I'm happy to assist with debugging and testing.
--
Petr Špaček
Internet Systems Consortium
^ permalink raw reply [flat|nested] 6+ messages in thread
* [bug] nsinfo__mountns_enter failure can cause perf to operate on wrong file
@ 2023-10-30 16:42 Petr Špaček
2023-10-30 19:40 ` Petr Špaček
0 siblings, 1 reply; 6+ messages in thread
From: Petr Špaček @ 2023-10-30 16:42 UTC (permalink / raw)
To: linux-perf-users
Hello,
I've noticed that sometimes "perf record" confuses paths between two
different namespaces and inadvertently operates on a wrong file (on the
same path - but wrong mount namespace).
Version: 6.5, commit 750b95887e567848ac2c851dae47922cac6db946 from
Linus's tree
Reproducer using Podman container:
All commands executed as non-privileged user, except the sysctl tweak.
$ sudo sysctl -w 'kernel.perf_event_paranoid = -1'
# binary in the container needs to have the same path as on host
# but different content
$ podman run -ti docker.io/library/ubuntu:23.10 /usr/bin/yes > /dev/null
# leave this running and get PID of the process inside the container
$ docker inspect $(docker ps --latest -q) | jq '.[0].State.Pid'
# ! record as an unprivileged user !
# record and use debuginfod from official Ubuntu servers
$ perf record --debuginfod='https://debuginfod.ubuntu.com/' --all-user
-F 99 --pid 637217
# check if we got symbols ... we should have because debuginfod has been
configured
$ perf script # notice "unknown" all over the place
# check build-ids in the recording
$ perf buildid-list
f53cbc885777b8cfc9e54a8015318a71e6845bc3 /usr/bin/yes
# check build-id in the container
$ docker exec -ti $(docker ps --latest -q) bash -c 'apt update; apt
install -yyy file; file `which yes`'
... BuildID[sha1]=7c895824831420bd30a372431d2b241bb6ff5554 ...
Values in perf output and in the containerized binary do not match. Huh?
Where the "perf" value came from, anyway?
Turns out the buildid-list shows value from the _host_:
$ file /usr/bin/yes
... BuildID[sha1]=f53cbc885777b8cfc9e54a8015318a71e6845bc3 ...
My guess is that this can happen when the user executing "perf record"
does not have privileges to nsenter() the target namespace, but this
failure is not checked.
I could not follow "perf record" code, but "perf buildid-cache --add"
calls nsinfo__mountns_enter() which has void return type, and hilarity
ensues if that call fails.
It works as expected if I add "sudo" in front of each "perf" command.
I'm happy to assist with debugging and testing.
--
Petr Špaček
Internet Systems Consortium
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [bug] nsinfo__mountns_enter failure can cause perf to operate on wrong file
2023-10-30 16:42 [bug] nsinfo__mountns_enter failure can cause perf to operate on wrong file Petr Špaček
@ 2023-10-30 19:40 ` Petr Špaček
0 siblings, 0 replies; 6+ messages in thread
From: Petr Špaček @ 2023-10-30 19:40 UTC (permalink / raw)
To: linux-perf-users
On 30. 10. 23 17:42, Petr Špaček wrote:
> Hello,
>
> I've noticed that sometimes "perf record" confuses paths between two
> different namespaces and inadvertently operates on a wrong file (on the
> same path - but wrong mount namespace).
Apologies for duplicates. For some reason the previous attempts did not
show up in my inbox.
--
Petr Špaček
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [bug] nsinfo__mountns_enter failure can cause perf to operate on wrong file
2023-10-27 15:58 Petr Špaček
@ 2023-11-06 4:32 ` Namhyung Kim
2023-11-20 9:28 ` Petr Špaček
0 siblings, 1 reply; 6+ messages in thread
From: Namhyung Kim @ 2023-11-06 4:32 UTC (permalink / raw)
To: Petr Špaček; +Cc: linux-perf-users
Hello,
Sorry for the late reply.
On Fri, Oct 27, 2023 at 8:58 AM Petr Špaček <pspacek@isc.org> wrote:
>
> Hello,
>
> I've noticed that sometimes "perf record" confuses paths between two
> different namespaces and inadvertently operates on a wrong file (on the
> same path - but wrong mount namespace).
>
> Version: 6.5, commit 750b95887e567848ac2c851dae47922cac6db946 from
> Linus's tree
>
> Reproducer using Podman container:
> All commands executed as non-privileged user, except the sysctl tweak.
>
> $ sudo sysctl -w 'kernel.perf_event_paranoid = -1'
>
> # binary in the container needs to have the same path as on host
> # but different content
> $ podman run -ti docker.io/library/ubuntu:23.10 /usr/bin/yes > /dev/null
>
> # leave this running and get PID of the process inside the container
> $ docker inspect $(docker ps --latest -q) | jq '.[0].State.Pid'
>
> # ! record as an unprivileged user !
> # record and use debuginfod from official Ubuntu servers
> $ perf record --debuginfod='https://debuginfod.ubuntu.com/' --all-user
> -F 99 --pid 637217
>
> # check if we got symbols ... we should have because debuginfod has been
> configured
> $ perf script # notice "unknown" all over the place
>
> # check build-ids in the recording
> $ perf buildid-list
> f53cbc885777b8cfc9e54a8015318a71e6845bc3 /usr/bin/yes
>
> # check build-id in the container
> $ docker exec -ti $(docker ps --latest -q) bash -c 'apt update; apt
> install -yyy file; file `which yes`'
> ... BuildID[sha1]=7c895824831420bd30a372431d2b241bb6ff5554 ...
>
>
> Values in perf output and in the containerized binary do not match. Huh?
> Where the "perf" value came from, anyway?
>
> Turns out the buildid-list shows value from the _host_:
> $ file /usr/bin/yes
> ... BuildID[sha1]=f53cbc885777b8cfc9e54a8015318a71e6845bc3 ...
>
>
> My guess is that this can happen when the user executing "perf record"
> does not have privileges to nsenter() the target namespace, but this
> failure is not checked.
>
> I could not follow "perf record" code, but "perf buildid-cache --add"
> calls nsinfo__mountns_enter() which has void return type, and hilarity
> ensues if that call fails.
Right, but I guess there's not much it can do in that case.
Maybe it can warn the user it could result in unknown symbols.
Thanks,
Namhyung
>
> It works as expected if I add "sudo" in front of each "perf" command.
>
> I'm happy to assist with debugging and testing.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [bug] nsinfo__mountns_enter failure can cause perf to operate on wrong file
2023-11-06 4:32 ` Namhyung Kim
@ 2023-11-20 9:28 ` Petr Špaček
0 siblings, 0 replies; 6+ messages in thread
From: Petr Špaček @ 2023-11-20 9:28 UTC (permalink / raw)
To: Namhyung Kim; +Cc: linux-perf-users
On 06. 11. 23 5:32, Namhyung Kim wrote:
> On Fri, Oct 27, 2023 at 8:58 AM Petr Špaček <pspacek@isc.org> wrote:
>> I've noticed that sometimes "perf record" confuses paths between two
>> different namespaces and inadvertently operates on a wrong file (on the
>> same path - but wrong mount namespace).
>>
>> Version: 6.5, commit 750b95887e567848ac2c851dae47922cac6db946 from
>> Linus's tree
... snip ...
>> Values in perf output and in the containerized binary do not match. Huh?
>> Where the "perf" value came from, anyway?
>>
>> Turns out the buildid-list shows value from the _host_:
>> $ file /usr/bin/yes
>> ... BuildID[sha1]=f53cbc885777b8cfc9e54a8015318a71e6845bc3 ...
>>
>>
>> My guess is that this can happen when the user executing "perf record"
>> does not have privileges to nsenter() the target namespace, but this
>> failure is not checked.
... snip ...
> Right, but I guess there's not much it can do in that case.
> Maybe it can warn the user it could result in unknown symbols.
From user perspective _adding wrong_ buildid is much worse than _not
adding_ buildid at all.
Wrong buildit messes up output even after moving the trace into correct
root filesystem (or fixing permissions).
Would it be possible to detect the error & skip adding build id in that
case, please?
I'm happy to assist with testing as needed. Thank you for your time.
--
Petr Špaček
Internet Systems Consortium
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-11-20 9:28 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-10-30 16:42 [bug] nsinfo__mountns_enter failure can cause perf to operate on wrong file Petr Špaček
2023-10-30 19:40 ` Petr Špaček
-- strict thread matches above, loose matches on Subject: below --
2023-10-27 15:58 Petr Špaček
2023-11-06 4:32 ` Namhyung Kim
2023-11-20 9:28 ` Petr Špaček
2023-10-27 15:27 Petr Špaček
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).