[PATCH] perf: fix race in build_id_cache__add_s()

[PATCH] perf: fix race in build_id_cache__add_s()

[Milos Vyletel]

int build_id_cache__add_s(const char *sbuild_id, const char *debugdir,
                          const char *name, bool is_kallsyms, bool is_vdso)
{
...
        if (access(filename, F_OK)) {
               ^--------------------------------------------------------- [1]
                if (is_kallsyms) {
                         if (copyfile("/proc/kallsyms", filename))
                                goto out_free;
                } else if (link(realname, filename) && copyfile(name, filename))
                             ^-----------------------------^------------- [2]
                                                            \------------ [3]
                        goto out_free;
        }
...

when multiple instances of perf record get to [1] at more or less same time and
run access() one or more may get failure because the file does not exist yet
(since the first instance did not have chance to link it yet). at this point the
race moves to link() at [2] where first thread to get there links file and goes
on but second one gets -EEXIST so it runs copyfile [3] which truncates the file.

recreator:
rm -rf /root/.debug
for cpu in $(awk '/processor/ {print $3}' /proc/cpuinfo); do
	perf record -a -v -T -F 1000 -C $cpu \
		-o perf-${cpu}.data sleep 5 2> /dev/null &
done
wait

and simply search for empty files by
find /lib/modules/`uname -r`/kernel/* -size 0

Signed-off-by: Milos Vyletel <milos@redhat.com>
---
 tools/perf/util/build-id.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index 0c72680..a9db5a1 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -293,7 +293,8 @@ int build_id_cache__add_s(const char *sbuild_id, const char *debugdir,
 		if (is_kallsyms) {
 			 if (copyfile("/proc/kallsyms", filename))
 				goto out_free;
-		} else if (link(realname, filename) && copyfile(name, filename))
+		} else if (link(realname, filename) && errno != EEXIST &&
+				copyfile(name, filename))
 			goto out_free;
 	}
 
-- 
2.1.0

Re: [PATCH] perf: fix race in build_id_cache__add_s()

[Jiri Olsa]

On Fri, Mar 20, 2015 at 11:37:25AM +0100, Milos Vyletel wrote:
> int build_id_cache__add_s(const char *sbuild_id, const char *debugdir,
>                           const char *name, bool is_kallsyms, bool is_vdso)
> {
> ...
>         if (access(filename, F_OK)) {
>                ^--------------------------------------------------------- [1]
>                 if (is_kallsyms) {
>                          if (copyfile("/proc/kallsyms", filename))
>                                 goto out_free;
>                 } else if (link(realname, filename) && copyfile(name, filename))
>                              ^-----------------------------^------------- [2]
>                                                             \------------ [3]
>                         goto out_free;
>         }
> ...
> 
> when multiple instances of perf record get to [1] at more or less same time and
> run access() one or more may get failure because the file does not exist yet
> (since the first instance did not have chance to link it yet). at this point the
> race moves to link() at [2] where first thread to get there links file and goes
> on but second one gets -EEXIST so it runs copyfile [3] which truncates the file.

nice.. :-\ 

Acked-by: Jiri Olsa <jolsa@kernel.org>

in addition we should use some inter-perf lock
on all .debug dir operations

jirka

Re: [PATCH] perf: fix race in build_id_cache__add_s()

[Arnaldo Carvalho de Melo]

Em Fri, Mar 20, 2015 at 01:18:07PM +0100, Jiri Olsa escreveu:
> On Fri, Mar 20, 2015 at 11:37:25AM +0100, Milos Vyletel wrote:
> > int build_id_cache__add_s(const char *sbuild_id, const char *debugdir,
> >                           const char *name, bool is_kallsyms, bool is_vdso)
> > {
> > ...
> >         if (access(filename, F_OK)) {
> >                ^--------------------------------------------------------- [1]
> >                 if (is_kallsyms) {
> >                          if (copyfile("/proc/kallsyms", filename))
> >                                 goto out_free;
> >                 } else if (link(realname, filename) && copyfile(name, filename))
> >                              ^-----------------------------^------------- [2]
> >                                                             \------------ [3]
> >                         goto out_free;
> >         }
> > ...
> > 
> > when multiple instances of perf record get to [1] at more or less same time and
> > run access() one or more may get failure because the file does not exist yet
> > (since the first instance did not have chance to link it yet). at this point the
> > race moves to link() at [2] where first thread to get there links file and goes
> > on but second one gets -EEXIST so it runs copyfile [3] which truncates the file.
> 
> nice.. :-\ 
> 
> Acked-by: Jiri Olsa <jolsa@kernel.org>
> 
> in addition we should use some inter-perf lock
> on all .debug dir operations

Yeah, would be nice to have that improved.

Thanks, applied.

- Arnaldo

[tip:perf/core] perf tools: Fix race in build_id_cache__add_s()

[tip-bot for Milos Vyletel]

Commit-ID:  0635b0f71424be7706793ac260d063491a2889a0
Gitweb:     http://git.kernel.org/tip/0635b0f71424be7706793ac260d063491a2889a0
Author:     Milos Vyletel <milos@redhat.com>
AuthorDate: Fri, 20 Mar 2015 11:37:25 +0100
Committer:  Arnaldo Carvalho de Melo <acme@redhat.com>
CommitDate: Fri, 20 Mar 2015 17:49:50 -0300

perf tools: Fix race in build_id_cache__add_s()

int build_id_cache__add_s(const char *sbuild_id, const char *debugdir,
                          const char *name, bool is_kallsyms, bool is_vdso)
{
...
        if (access(filename, F_OK)) {
               ^--------------------------------------------------------- [1]
                if (is_kallsyms) {
                         if (copyfile("/proc/kallsyms", filename))
                                goto out_free;
                } else if (link(realname, filename) && copyfile(name, filename))
                             ^-----------------------------^------------- [2]
                                                            \------------ [3]
                        goto out_free;
        }
...

When multiple instances of perf record get to [1] at more or less same time and
run access() one or more may get failure because the file does not exist yet
(since the first instance did not have chance to link it yet).

At this point the race moves to link() at [2] where first thread to get
there links file and goes on but second one gets -EEXIST so it runs
copyfile [3] which truncates the file.

reproducer:

rm -rf /root/.debug
for cpu in $(awk '/processor/ {print $3}' /proc/cpuinfo); do
	perf record -a -v -T -F 1000 -C $cpu \
		-o perf-${cpu}.data sleep 5 2> /dev/null &
done
wait

and simply search for empty files by:

find /lib/modules/`uname -r`/kernel/* -size 0

Signed-off-by: Milos Vyletel <milos@redhat.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1426847846-11112-1-git-send-email-milos@redhat.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/util/build-id.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/build-id.c b/tools/perf/util/build-id.c
index a196746..f7fb258 100644
--- a/tools/perf/util/build-id.c
+++ b/tools/perf/util/build-id.c
@@ -374,7 +374,8 @@ int build_id_cache__add_s(const char *sbuild_id, const char *name,
 		if (is_kallsyms) {
 			 if (copyfile("/proc/kallsyms", filename))
 				goto out_free;
-		} else if (link(realname, filename) && copyfile(name, filename))
+		} else if (link(realname, filename) && errno != EEXIST &&
+				copyfile(name, filename))
 			goto out_free;
 	}