* Intermittent perf build failures
@ 2016-10-11 20:43 Laura Abbott
2016-10-11 20:59 ` Jiri Olsa
0 siblings, 1 reply; 6+ messages in thread
From: Laura Abbott @ 2016-10-11 20:43 UTC (permalink / raw)
To: Jiri Olsa, Andi Kleen, Arnaldo Carvalho de Melo,
Arnaldo Carvalho de Melo
Cc: Peter Zijlstra, Linux Kernel Mailing List, Ingo Molnar
Hi,
While building today's Fedora rawhide kernel, there was a failure
building perf with -j4 [1]:
gcc -Wp,-MD,./.plugin_scsi.o.d,-MT,plugin_scsi.o -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -D"BUILD_STR(s)=#s" -c -o plugin_scsi.o plugin_scsi.c
ld -r -o plugin_function-in.o plugin_function.o
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_jbd2.so plugin_jbd2-in.o
gcc -Wp,-MD,./.plugin_cfg80211.o.d,-MT,plugin_cfg80211.o -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -D"BUILD_STR(s)=#s" -c -o plugin_cfg80211.o plugin_cfg80211.c
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_hrtimer.so plugin_hrtimer-in.o
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_kmem.so plugin_kmem-in.o
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_kvm.so plugin_kvm-in.o
ld -r -o plugin_xen-in.o plugin_xen.o
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_mac80211.so plugin_mac80211-in.o
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_sched_switch.so plugin_sched_switch-in.o
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_function.so plugin_function-in.o
ld -r -o plugin_cfg80211-in.o plugin_cfg80211.o
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_xen.so plugin_xen-in.o
gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_cfg80211.so plugin_cfg80211-in.o
gcc -Wp,-MD,./.plugin_scsi.o.d,-MT,plugin_scsi.o -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -D"BUILD_STR(s)=#s" -c -o plugin_scsi.o plugin_scsi.c
ld -r -o plugin_scsi-in.o plugin_scsi.o
plugin_scsi.o: file not recognized: File truncated
make[3]: *** [/builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/build/Makefile.build:140: plugin_scsi-in.o] Error 1
make[2]: *** [Makefile:189: plugin_scsi-in.o] Error 2
make[1]: *** [Makefile.perf:570: libtraceevent_plugins] Error 2
make: *** [Makefile:109: install-traceevent-plugins] Error 2
The gcc call to compile plugin_scsi.o seems to be invoked twice which would
account for the ld failing. Unfortunately, this failure seems to be
intermittent and I've only been able to reproduce it once outside of
the Fedora build environment so bisection doesn't seem feasible.
The build invocation for this was changed in
0c3b7e42616f ("tools build: Add support for host programs format") so
I'm suspicious of that commit. I'm reverting that commit for today's
build but I'm not sure how much that's going to indicate given the
intermittent nature of the failure.
Any ideas?
Thanks,
Laura
[1] https://kojipkgs.fedoraproject.org//work/tasks/415/16050415/build.log
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Intermittent perf build failures
2016-10-11 20:43 Intermittent perf build failures Laura Abbott
@ 2016-10-11 20:59 ` Jiri Olsa
[not found] ` <0babd89f-dd38-89f5-7073-6b74c6692995@redhat.com>
0 siblings, 1 reply; 6+ messages in thread
From: Jiri Olsa @ 2016-10-11 20:59 UTC (permalink / raw)
To: Laura Abbott
Cc: Andi Kleen, Arnaldo Carvalho de Melo, Peter Zijlstra,
Linux Kernel Mailing List, Ingo Molnar
On Tue, Oct 11, 2016 at 01:43:36PM -0700, Laura Abbott wrote:
> Hi,
>
> While building today's Fedora rawhide kernel, there was a failure
> building perf with -j4 [1]:
>
>
> gcc -Wp,-MD,./.plugin_scsi.o.d,-MT,plugin_scsi.o -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -D"BUILD_STR(s)=#s" -c -o plugin_scsi.o plugin_scsi.c
> ld -r -o plugin_function-in.o plugin_function.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_jbd2.so plugin_jbd2-in.o
> gcc -Wp,-MD,./.plugin_cfg80211.o.d,-MT,plugin_cfg80211.o -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -D"BUILD_STR(s)=#s" -c -o plugin_cfg80211.o plugin_cfg80211.c
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_hrtimer.so plugin_hrtimer-in.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_kmem.so plugin_kmem-in.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_kvm.so plugin_kvm-in.o
> ld -r -o plugin_xen-in.o plugin_xen.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_mac80211.so plugin_mac80211-in.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_sched_switch.so plugin_sched_switch-in.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_function.so plugin_function-in.o
> ld -r -o plugin_cfg80211-in.o plugin_cfg80211.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_xen.so plugin_xen-in.o
> gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -shared -nostartfiles -o plugin_cfg80211.so plugin_cfg80211-in.o
> gcc -Wp,-MD,./.plugin_scsi.o.d,-MT,plugin_scsi.o -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fPIC -I. -I /builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/include '-DPLUGIN_DIR="/usr/lib64/traceevent/plugins"' -D_GNU_SOURCE -D"BUILD_STR(s)=#s" -c -o plugin_scsi.o plugin_scsi.c
> ld -r -o plugin_scsi-in.o plugin_scsi.o
> plugin_scsi.o: file not recognized: File truncated
> make[3]: *** [/builddir/build/BUILD/kernel-4.8.fc26/linux-4.9.0-0.rc0.git6.1.fc26.aarch64/tools/build/Makefile.build:140: plugin_scsi-in.o] Error 1
> make[2]: *** [Makefile:189: plugin_scsi-in.o] Error 2
> make[1]: *** [Makefile.perf:570: libtraceevent_plugins] Error 2
> make: *** [Makefile:109: install-traceevent-plugins] Error 2
>
> The gcc call to compile plugin_scsi.o seems to be invoked twice which would
> account for the ld failing. Unfortunately, this failure seems to be
> intermittent and I've only been able to reproduce it once outside of
> the Fedora build environment so bisection doesn't seem feasible.
hi,
yep, looks like build race.. but -j4 should not be that bad ;-)
some questions:
- how often do you see this?
- is it always plugin_scsi.o failing?
- hows the build invoked?
- could you please attach full build log?
thanks,
jirka
>
> The build invocation for this was changed in
> 0c3b7e42616f ("tools build: Add support for host programs format") so
> I'm suspicious of that commit. I'm reverting that commit for today's
> build but I'm not sure how much that's going to indicate given the
> intermittent nature of the failure.
>
> Any ideas?
>
> Thanks,
> Laura
>
>
> [1] https://kojipkgs.fedoraproject.org//work/tasks/415/16050415/build.log
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-10-16 15:01 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-10-11 20:43 Intermittent perf build failures Laura Abbott
2016-10-11 20:59 ` Jiri Olsa
[not found] ` <0babd89f-dd38-89f5-7073-6b74c6692995@redhat.com>
2016-10-12 10:12 ` Jiri Olsa
2016-10-12 18:18 ` Laura Abbott
2016-10-16 7:14 ` Ingo Molnar
2016-10-16 15:01 ` Jiri Olsa
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).