Parallel build failures with util-linux 2.40

Parallel build failures with util-linux 2.40

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 2417 bytes --]

Hi,

Since updating from util-linux 2.39.3 to 2.40.1 in Nixpkgs, we've been
seeing intermittent parallel build failures like the following:

libtool: warning(B: relinking 'pam_lastlog2.la'(B
libtool: install: (cd /build/util-linux-2.40.1; /nix/store/306znyj77fv49kwnkpxmb0j2znqpa8bj-bash-5.2p26/bin/bash "/build/util-linux-2.40.1/libtool"  --silent --tag CC --mode=relink gcc -fsigned-char -fno-common -Wall -Wextra -Waddress-of-packed-member -Wdiscarded-qualifiers -Wformat-security -Wimplicit-function-declaration -Wmissing-declarations -Wmissing-parameter-type -Wmissing-prototypes -Wnested-externs -Wno-missing-field-initializers -Wold-style-definition -Wpointer-arith -Wredundant-decls -Wsign-compare -Wstrict-prototypes -Wtype-limits -Wuninitialized -Wunused-but-set-parameter -Wunused-but-set-variable -Wunused-parameter -Wunused-result -Wunused-variable -Wvla -Walloca -Werror=sequence-point -I./liblastlog2/src -g -O2 -module -avoid-version -shared pam_lastlog2_la_LDFLAGS += -Wl,--version-script,./pam_lastlog2/src/pam_lastlog2.sym -o pam_lastlog2.la -rpath /nix/store/ri9wphsskq6ygxigvlbahb9mz379qiz5-util-linux-2.40.1-lib/lib/security pam_lastlog2/src/la-pam_lastlog2.lo liblastlog2.la )
libtool: warning(B: 'libmount.la' has not been installed in '/nix/store/ri9wphsskq6ygxigvlbahb9mz379qiz5-util-linux-2.40.1-lib/lib'(B
libtool: warning(B: '/build/util-linux-2.40.1/libblkid.la' has not been installed in '/nix/store/ri9wphsskq6ygxigvlbahb9mz379qiz5-util-linux-2.40.1-lib/lib'(B
libtool: install: /nix/store/php4qidg2bxzmm79vpri025bqi0fa889-coreutils-9.5/bin/install -c .libs/mount /nix/store/v6ff7704fchg30kj4z422693lgnnmmbn-util-linux-2.40.1-bin/bin/mount
/nix/store/bgcaxhhxswzvmxjbbgvvaximm5hwghz1-binutils-2.41/bin/ld: cannot find -llastlog2: No such file or directory
collect2: error: ld returned 1 exit status
libtool: warning(B: 'libmount.la' has not been installed in '/nix/store/ri9wphsskq6ygxigvlbahb9mz379qiz5-util-linux-2.40.1-lib/lib'(B
libtool: warning(B: '/build/util-linux-2.40.1/libblkid.la' has not been installed in '/nix/store/ri9wphsskq6ygxigvlbahb9mz379qiz5-util-linux-2.40.1-lib/lib'(B
libtool:   error(B: error: relink 'pam_lastlog2.la' with the above command before installing it(B
make[3]: *** [Makefile:7598: install-securelibLTLIBRARIES] Error 1

Full log:
https://cache.nixos.org/log/pp817g4wf18pn7hrn4a7377yxf06bprw-util-linux-2.40.1.drv

Alyssa Ross

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Parallel build failures with util-linux 2.40

[Karel Zak]

On Tue, May 14, 2024 at 12:01:23PM +0200, Alyssa Ross wrote:
> Since updating from util-linux 2.39.3 to 2.40.1 in Nixpkgs, we've been
> seeing intermittent parallel build failures like the following:

What does it mean "parallel build"? make -j ?

> libtool: warning(B: relinking 'pam_lastlog2.la'(B libtool: install:
> (cd /build/util-linux-2.40.1;
> /nix/store/306znyj77fv49kwnkpxmb0j2znqpa8bj-bash-5.2p26/bin/bash
> "/build/util-linux-2.40.1/libtool"  --silent --tag CC --mode=relink
> gcc -fsigned-char -fno-common -Wall -Wextra
> -Waddress-of-packed-member -Wdiscarded-qualifiers -Wformat-security
> -Wimplicit-function-declaration -Wmissing-declarations
> -Wmissing-parameter-type -Wmissing-prototypes -Wnested-externs
> -Wno-missing-field-initializers -Wold-style-definition
> -Wpointer-arith -Wredundant-decls -Wsign-compare -Wstrict-prototypes
> -Wtype-limits -Wuninitialized -Wunused-but-set-parameter
> -Wunused-but-set-variable -Wunused-parameter -Wunused-result
> -Wunused-variable -Wvla -Walloca -Werror=sequence-point
> -I./liblastlog2/src -g -O2 -module -avoid-version -shared
> pam_lastlog2_la_LDFLAGS +=

I'm not sure, but it seems like you're being affected by an extra "+="
in the LDFLAGS. This should be fixed by...

    https://github.com/util-linux/util-linux/commit/290748729dc3edf9ea1c680c8954441a5e367a44
    https://github.com/util-linux/util-linux/commit/597e8b246ae31366514ead6cca240a09fe5e1528

Can you try your use case with the latest git tree?

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: Parallel build failures with util-linux 2.40

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 1854 bytes --]

Karel Zak <kzak@redhat.com> writes:

> On Tue, May 14, 2024 at 12:01:23PM +0200, Alyssa Ross wrote:
>> Since updating from util-linux 2.39.3 to 2.40.1 in Nixpkgs, we've been
>> seeing intermittent parallel build failures like the following:
>
> What does it mean "parallel build"? make -j ?

Yeah.

>> libtool: warning(B: relinking 'pam_lastlog2.la'(B libtool: install:
>> (cd /build/util-linux-2.40.1;
>> /nix/store/306znyj77fv49kwnkpxmb0j2znqpa8bj-bash-5.2p26/bin/bash
>> "/build/util-linux-2.40.1/libtool"  --silent --tag CC --mode=relink
>> gcc -fsigned-char -fno-common -Wall -Wextra
>> -Waddress-of-packed-member -Wdiscarded-qualifiers -Wformat-security
>> -Wimplicit-function-declaration -Wmissing-declarations
>> -Wmissing-parameter-type -Wmissing-prototypes -Wnested-externs
>> -Wno-missing-field-initializers -Wold-style-definition
>> -Wpointer-arith -Wredundant-decls -Wsign-compare -Wstrict-prototypes
>> -Wtype-limits -Wuninitialized -Wunused-but-set-parameter
>> -Wunused-but-set-variable -Wunused-parameter -Wunused-result
>> -Wunused-variable -Wvla -Walloca -Werror=sequence-point
>> -I./liblastlog2/src -g -O2 -module -avoid-version -shared
>> pam_lastlog2_la_LDFLAGS +=
>
> I'm not sure, but it seems like you're being affected by an extra "+="
> in the LDFLAGS. This should be fixed by...
>
>     https://github.com/util-linux/util-linux/commit/290748729dc3edf9ea1c680c8954441a5e367a44
>     https://github.com/util-linux/util-linux/commit/597e8b246ae31366514ead6cca240a09fe5e1528
>
> Can you try your use case with the latest git tree?

Thanks for pointing these out — I've applied the patches to our build,
but since the failure only happened intermittently, I'm not going to be
able to tell if it's fixed the issue right away.  We'll just have to
wait and see, and no news will be good news.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Parallel build failures with util-linux 2.40

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 5917 bytes --]

Alyssa Ross <hi@alyssa.is> writes:

> Karel Zak <kzak@redhat.com> writes:
>
>> On Tue, May 14, 2024 at 12:01:23PM +0200, Alyssa Ross wrote:
>>> Since updating from util-linux 2.39.3 to 2.40.1 in Nixpkgs, we've been
>>> seeing intermittent parallel build failures like the following:
>>
>> What does it mean "parallel build"? make -j ?
>
> Yeah.
>
>>> libtool: warning(B: relinking 'pam_lastlog2.la'(B libtool: install:
>>> (cd /build/util-linux-2.40.1;
>>> /nix/store/306znyj77fv49kwnkpxmb0j2znqpa8bj-bash-5.2p26/bin/bash
>>> "/build/util-linux-2.40.1/libtool"  --silent --tag CC --mode=relink
>>> gcc -fsigned-char -fno-common -Wall -Wextra
>>> -Waddress-of-packed-member -Wdiscarded-qualifiers -Wformat-security
>>> -Wimplicit-function-declaration -Wmissing-declarations
>>> -Wmissing-parameter-type -Wmissing-prototypes -Wnested-externs
>>> -Wno-missing-field-initializers -Wold-style-definition
>>> -Wpointer-arith -Wredundant-decls -Wsign-compare -Wstrict-prototypes
>>> -Wtype-limits -Wuninitialized -Wunused-but-set-parameter
>>> -Wunused-but-set-variable -Wunused-parameter -Wunused-result
>>> -Wunused-variable -Wvla -Walloca -Werror=sequence-point
>>> -I./liblastlog2/src -g -O2 -module -avoid-version -shared
>>> pam_lastlog2_la_LDFLAGS +=
>>
>> I'm not sure, but it seems like you're being affected by an extra "+="
>> in the LDFLAGS. This should be fixed by...
>>
>>     https://github.com/util-linux/util-linux/commit/290748729dc3edf9ea1c680c8954441a5e367a44
>>     https://github.com/util-linux/util-linux/commit/597e8b246ae31366514ead6cca240a09fe5e1528
>>
>> Can you try your use case with the latest git tree?
>
> Thanks for pointing these out — I've applied the patches to our build,
> but since the failure only happened intermittently, I'm not going to be
> able to tell if it's fixed the issue right away.  We'll just have to
> wait and see, and no news will be good news.

Hi, I'm afraid we're still seeing similar failures with those two
patches backported on top of 2.40.1:

	  /nix/store/3c54hwp1zji0h8yl66860cp8zp2vsrzv-bash-5.2p26/bin/bash ./libtool   --mode=install /nix/store/dm0wz420s9n1nbp3731y9iqya0zswyy8-coreutils-9.5/bin/install -c login dmesg mount umount wdctl mountpoint lsblk findmnt kill lsfd pipesz '/nix/store/hxd66fx3rvq1km5hjq6dgx8a6yw7czyd-util-linux-minimal-2.40.1-bin/bin'
	libtool: warning: relinking 'pam_lastlog2.la'
	libtool: install: (cd /build/util-linux-2.40.1; /nix/store/3c54hwp1zji0h8yl66860cp8zp2vsrzv-bash-5.2p26/bin/bash "/build/util-linux-2.40.1/libtool"  --silent --tag CC --mode=relink gcc -fsigned-char -fno-common -Wall -Wextra -Waddress-of-packed-member -Wdiscarded-qualifiers -Wformat-security -Wimplicit-function-declaration -Wmissing-declarations -Wmissing-parameter-type -Wmissing-prototypes -Wnested-externs -Wno-missing-field-initializers -Wold-style-definition -Wpointer-arith -Wredundant-decls -Wsign-compare -Wstrict-prototypes -Wtype-limits -Wuninitialized -Wunused-but-set-parameter -Wunused-but-set-variable -Wunused-parameter -Wunused-result -Wunused-variable -Wvla -Walloca -Werror=sequence-point -I./liblastlog2/src -g -O2 -module -avoid-version -shared -Wl,--version-script,./pam_lastlog2/src/pam_lastlog2.sym -o pam_lastlog2.la -rpath /nix/store/dsxjv3a5aj1f4pifv5a6rvpj7zk00vyx-util-linux-minimal-2.40.1-lib/lib/security pam_lastlog2/src/la-pam_lastlog2.lo liblastlog2.la )
	libtool: install: /nix/store/dm0wz420s9n1nbp3731y9iqya0zswyy8-coreutils-9.5/bin/install -c login /nix/store/hxd66fx3rvq1km5hjq6dgx8a6yw7czyd-util-linux-minimal-2.40.1-bin/bin/login
	libtool: install: /nix/store/dm0wz420s9n1nbp3731y9iqya0zswyy8-coreutils-9.5/bin/install -c dmesg /nix/store/hxd66fx3rvq1km5hjq6dgx8a6yw7czyd-util-linux-minimal-2.40.1-bin/bin/dmesg
	libtool: warning: 'libmount.la' has not been installed in '/nix/store/dsxjv3a5aj1f4pifv5a6rvpj7zk00vyx-util-linux-minimal-2.40.1-lib/lib'
	libtool: warning: '/build/util-linux-2.40.1/libblkid.la' has not been installed in '/nix/store/dsxjv3a5aj1f4pifv5a6rvpj7zk00vyx-util-linux-minimal-2.40.1-lib/lib'
	libtool: install: /nix/store/dm0wz420s9n1nbp3731y9iqya0zswyy8-coreutils-9.5/bin/install -c .libs/mount /nix/store/hxd66fx3rvq1km5hjq6dgx8a6yw7czyd-util-linux-minimal-2.40.1-bin/bin/mount
	libtool: warning: 'libmount.la' has not been installed in '/nix/store/dsxjv3a5aj1f4pifv5a6rvpj7zk00vyx-util-linux-minimal-2.40.1-lib/lib'
	libtool: warning: '/build/util-linux-2.40.1/libblkid.la' has not been installed in '/nix/store/dsxjv3a5aj1f4pifv5a6rvpj7zk00vyx-util-linux-minimal-2.40.1-lib/lib'
	libtool: install: /nix/store/dm0wz420s9n1nbp3731y9iqya0zswyy8-coreutils-9.5/bin/install -c .libs/umount /nix/store/hxd66fx3rvq1km5hjq6dgx8a6yw7czyd-util-linux-minimal-2.40.1-bin/bin/umount
	libtool: warning: 'libsmartcols.la' has not been installed in '/nix/store/dsxjv3a5aj1f4pifv5a6rvpj7zk00vyx-util-linux-minimal-2.40.1-lib/lib'
	libtool: install: /nix/store/dm0wz420s9n1nbp3731y9iqya0zswyy8-coreutils-9.5/bin/install -c .libs/wdctl /nix/store/hxd66fx3rvq1km5hjq6dgx8a6yw7czyd-util-linux-minimal-2.40.1-bin/bin/wdctl
	/nix/store/8qjkvnam69dm0r9lhirxy1ngzbsh6a8z-binutils-2.41/bin/ld: cannot find -llastlog2: No such file or directory
	libtool: warning: 'libmount.la' has not been installed in '/nix/store/dsxjv3a5aj1f4pifv5a6rvpj7zk00vyx-util-linux-minimal-2.40.1-lib/lib'
	collect2: error: ld returned 1 exit status
	libtool: warning: '/build/util-linux-2.40.1/libblkid.la' has not been installed in '/nix/store/dsxjv3a5aj1f4pifv5a6rvpj7zk00vyx-util-linux-minimal-2.40.1-lib/lib'
	libtool:   error: error: relink 'pam_lastlog2.la' with the above command before installing it
	make[3]: *** [Makefile:7603: install-securelibLTLIBRARIES] Error 1

Full log:
https://hydra.nixos.org/build/260133082/nixlog/11/raw

I wish I could provide more helpful information, but since it doesn't
happen consistently it's hard to collect any.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Parallel build failures with util-linux 2.40

[Santiago Vila]

Hello.

In some cases, make --shuffle option (available in GNU make 4.4)
"amplifies" the probability that a Makefile bug shows up.
I suggest that you people give it a try.

There is a blog entry from the author explaining --shuffle option:

https://trofi.github.io/posts/238-new-make-shuffle-mode.html

[ I'm just a long-time lurker in this list, but maybe this could help ]

Thanks.

Re: Parallel build failures with util-linux 2.40

[Alyssa Ross]

[-- Attachment #1: Type: text/plain, Size: 581 bytes --]

Santiago Vila <sanvila@debian.org> writes:

> Hello.
>
> In some cases, make --shuffle option (available in GNU make 4.4)
> "amplifies" the probability that a Makefile bug shows up.
> I suggest that you people give it a try.
>
> There is a blog entry from the author explaining --shuffle option:
>
> https://trofi.github.io/posts/238-new-make-shuffle-mode.html
>
> [ I'm just a long-time lurker in this list, but maybe this could help ]

Thanks for the suggestion, but I tried running with --shuffle for hours
before submitting my report, but wasn't able to reproduce that way. :(

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]