native sysroot race with rpmbuild

native sysroot race with rpmbuild

[Jussi Kukkonen]

[-- Attachment #1: Type: text/plain, Size: 1301 bytes --]

I'm seeing lots of this failure in my last build:

| DEBUG: PKGWRITEDIR:
/mnt/extra-ssd/tmp/work/all-poky-linux/adwaita-icon-theme/3.24.0-r0/deploy-rpms/noarch
| DEBUG: SITE files ['endian-little', 'bit-32', 'common-linux',
'common-glibc', 'allarch-linux', 'common']
| DEBUG: Executing shell function BUILDSPEC
|
/mnt/extra-ssd/tmp/work/all-poky-linux/adwaita-icon-theme/3.24.0-r0/recipe-sysroot-native/usr/bin/rpmbuild:
error while loading shared libraries: libdw.so.1: cannot open shared object
file: No such file or directory
| WARNING: exit code 127 from a shell command.
| DEBUG: Python function do_package_rpm finished
| DEBUG: Python function do_package_write_rpm finished
| ERROR: Function failed: BUILDSPEC (log file is located at
/mnt/extra-ssd/tmp/work/all-poky-linux/adwaita-icon-theme/3.24.0-r0/temp/log.do_package_write_rpm.21827)
ERROR: Task
(/home/jku/src/poky/meta/recipes-gnome/gnome/adwaita-icon-theme_3.24.0.bb:do_package_write_rpm)
failed with exit code '1'

* libdw.so.1 isn't found when rpmbuild runs
* seems to be a race as the library is there when I check (and building
again works)
* Seems to mostly happen with allarch recipes (maybe they're just fast
enough): sysvinit-inittab, adwaita, xcursor-transparent-theme, iso-codes

Any ideas?

Jussi

[-- Attachment #2: Type: text/html, Size: 1569 bytes --]

Re: native sysroot race with rpmbuild

[Richard Purdie]

On Mon, 2017-08-14 at 11:33 +0300, Jussi Kukkonen wrote:
> I'm seeing lots of this failure in my last build:
> 
> | DEBUG: PKGWRITEDIR: /mnt/extra-ssd/tmp/work/all-poky-linux/adwaita-
> icon-theme/3.24.0-r0/deploy-rpms/noarch
> | DEBUG: SITE files ['endian-little', 'bit-32', 'common-linux',
> 'common-glibc', 'allarch-linux', 'common']
> | DEBUG: Executing shell function BUILDSPEC
> | /mnt/extra-ssd/tmp/work/all-poky-linux/adwaita-icon-theme/3.24.0-
> r0/recipe-sysroot-native/usr/bin/rpmbuild: error while loading shared
> libraries: libdw.so.1: cannot open shared object file: No such file
> or directory
> | WARNING: exit code 127 from a shell command.
> | DEBUG: Python function do_package_rpm finished
> | DEBUG: Python function do_package_write_rpm finished
> | ERROR: Function failed: BUILDSPEC (log file is located at
> /mnt/extra-ssd/tmp/work/all-poky-linux/adwaita-icon-theme/3.24.0-
> r0/temp/log.do_package_write_rpm.21827)
> ERROR: Task (/home/jku/src/poky/meta/recipes-gnome/gnome/adwaita-
> icon-theme_3.24.0.bb:do_package_write_rpm) failed with exit code '1'
> 
> * libdw.so.1 isn't found when rpmbuild runs
> * seems to be a race as the library is there when I check (and
> building again works)
> * Seems to mostly happen with allarch recipes (maybe they're just
> fast enough): sysvinit-inittab, adwaita, xcursor-transparent-theme,
> iso-codes
> 
> Any ideas?

Could you tar up /mnt/extra-ssd/tmp/work/all-poky-linux/adwaita-icon-
theme/3.24.0-r0/temp/log* and share somewhere please?

Not sure what is going on there, it seems odd :( What other tasks were
running when this happened?

I had a go at reproducing this locally but couldn't.

Cheers,

Richard

Re: native sysroot race with rpmbuild

[Jussi Kukkonen]

[-- Attachment #1: Type: text/plain, Size: 2247 bytes --]

On 14 August 2017 at 13:38, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:
>
> On Mon, 2017-08-14 at 11:33 +0300, Jussi Kukkonen wrote:
> > I'm seeing lots of this failure in my last build:
> >
> > | DEBUG: PKGWRITEDIR: /mnt/extra-ssd/tmp/work/all-poky-linux/adwaita-
> > icon-theme/3.24.0-r0/deploy-rpms/noarch
> > | DEBUG: SITE files ['endian-little', 'bit-32', 'common-linux',
> > 'common-glibc', 'allarch-linux', 'common']
> > | DEBUG: Executing shell function BUILDSPEC
> > | /mnt/extra-ssd/tmp/work/all-poky-linux/adwaita-icon-theme/3.24.0-
> > r0/recipe-sysroot-native/usr/bin/rpmbuild: error while loading shared
> > libraries: libdw.so.1: cannot open shared object file: No such file
> > or directory
> > | WARNING: exit code 127 from a shell command.
> > | DEBUG: Python function do_package_rpm finished
> > | DEBUG: Python function do_package_write_rpm finished
> > | ERROR: Function failed: BUILDSPEC (log file is located at
> > /mnt/extra-ssd/tmp/work/all-poky-linux/adwaita-icon-theme/3.24.0-
> > r0/temp/log.do_package_write_rpm.21827)
> > ERROR: Task (/home/jku/src/poky/meta/recipes-gnome/gnome/adwaita-
> > icon-theme_3.24.0.bb:do_package_write_rpm) failed with exit code '1'
> >
> > * libdw.so.1 isn't found when rpmbuild runs
> > * seems to be a race as the library is there when I check (and
> > building again works)
> > * Seems to mostly happen with allarch recipes (maybe they're just
> > fast enough): sysvinit-inittab, adwaita, xcursor-transparent-theme,
> > iso-codes
> >
> > Any ideas?
>
> Could you tar up /mnt/extra-ssd/tmp/work/all-poky-linux/adwaita-icon-
> theme/3.24.0-r0/temp/log* and share somewhere please?


Adwaita I had lost already but I've put the logs for failing
packagegroup-core-x11-sato here:
https://github.com/jku/packagegroup-core-x11-sato-logs

There's this in the log:
| NOTE: rpm-native exists in sysroot, skipping

>
> Not sure what is going on there, it seems odd :( What other tasks were
> running when this happened?

I was building core-image-sato so possibly many things.

>
> I had a go at reproducing this locally but couldn't.

It seemed to happen to lots of recipes but I can't reproduce afterwards
either.

Jussi

[-- Attachment #2: Type: text/html, Size: 2850 bytes --]

Re: native sysroot race with rpmbuild

[Richard Purdie]

On Mon, 2017-08-14 at 13:55 +0300, Jussi Kukkonen wrote:
> Adwaita I had lost already but I've put the logs for failing
> packagegroup-core-x11-sato here:
> https://github.com/jku/packagegroup-core-x11-sato-logs
>  
> There's this in the log:
> | NOTE: rpm-native exists in sysroot, skipping

What is really interesting is:

https://github.com/jku/packagegroup-core-x11-sato-logs/blob/master/log.do_package_qa.28854

which looks to have been running about the same time as:

https://github.com/jku/packagegroup-core-x11-sato-logs/blob/master/log.do_package_write_rpm.28850

(similar pids and listed in https://github.com/jku/packagegroup-core-x11-sato-logs/blob/master/log.task_order after one another)

Note that package_qa seems to think the dependency disappeared and is
uninstalling elfutils (which provides libdw) from the sysroot.

That might be enough to figure out some kind of reproducer. All very
interesting and "potentially problematic"...

Cheers,

Richard

Re: native sysroot race with rpmbuild

[Richard Purdie]

On Mon, 2017-08-14 at 11:33 +0300, Jussi Kukkonen wrote:
> I'm seeing lots of this failure in my last build:
> 
> | DEBUG: PKGWRITEDIR: /mnt/extra-ssd/tmp/work/all-poky-linux/adwaita-
> icon-theme/3.24.0-r0/deploy-rpms/noarch
> | DEBUG: SITE files ['endian-little', 'bit-32', 'common-linux',
> 'common-glibc', 'allarch-linux', 'common']
> | DEBUG: Executing shell function BUILDSPEC
> | /mnt/extra-ssd/tmp/work/all-poky-linux/adwaita-icon-theme/3.24.0-
> r0/recipe-sysroot-native/usr/bin/rpmbuild: error while loading shared
> libraries: libdw.so.1: cannot open shared object file: No such file
> or directory
> | WARNING: exit code 127 from a shell command.
> | DEBUG: Python function do_package_rpm finished
> | DEBUG: Python function do_package_write_rpm finished
> | ERROR: Function failed: BUILDSPEC (log file is located at
> /mnt/extra-ssd/tmp/work/all-poky-linux/adwaita-icon-theme/3.24.0-
> r0/temp/log.do_package_write_rpm.21827)
> ERROR: Task (/home/jku/src/poky/meta/recipes-gnome/gnome/adwaita-
> icon-theme_3.24.0.bb:do_package_write_rpm) failed with exit code '1'

I've just sent a patch to staging.bbclass which I believe should fix
this error. We also saw it on the autobuilder and I had other reports
about it.

Cheers,

Richard