Re: [OE-core] [PATCH v11] util-linux: split uuid in separate recipe to allow bootstrapping

["Luca Boccassi" <luca.boccassi@microsoft.com>]

On Wed, 2021-03-24 at 09:52 -0700, Scott Branden wrote:
> I have not debugged yet, but I suspect this change is causing the following failure if I merge the latest poky into our builds:
> 
> ERROR: mc:host:util-linux-uuid-2.36.2-r0 do_package: QA Issue: util-linux-uuid: Files/directories were installed but not shipped in any package:
>   /usr/lib/debug
>   /usr/lib/debug/usr
>   /usr/lib/debug/usr/lib
>   /usr/lib/debug/usr/lib/libuuid.so.1.3.0.debug
> Please set FILES such that these items are packaged. Alternatively if they are unneeded, avoid installing them or delete them within do_install.
> util-linux-uuid: 4 installed and not shipped files. [installed-vs-shipped]
> ERROR: mc:host:util-linux-uuid-2.36.2-r0 do_package: Fatal QA errors found, failing task.
> ERROR: Logfile of failure stored in: /hdd/yocto/genx/poky/build/tmp/work/core2-64-poky-linux/util-linux-uuid/2.36.2-r0/temp/log.do_package.31753
> ERROR: Task (mc:host:/hdd/yocto/genx/poky/build/../meta/recipes-core/util-linux/util-linux-uuid_2.36.2.bb:do_package) failed with exit code '1'

It most certainly is - is /usr/lib/debug configurable or supported?
Currently we have:

FILES_util-linux-libuuid-dbg = "/usr/src ${libdir}/.debug"

I'll send a patch to extend it and include /usr/lib/debug too

> On 2021-03-15 2:51 p.m., Richard Purdie wrote:
> > On Mon, 2021-03-15 at 13:57 +0000, Richard Purdie via lists.openembedded.org wrote:
> > > On Mon, 2021-03-15 at 14:55 +0100, Martin Jansa wrote:
> > > > On Mon, Mar 15, 2021 at 12:21:37PM +0000, Richard Purdie wrote:
> > > > > On Mon, 2021-03-15 at 11:50 +0000, Luca Boccassi wrote:
> > > > > > On Mon, 2021-03-15 at 10:49 +0000, Richard Purdie wrote:
> > > > > > > On Mon, 2021-03-15 at 10:44 +0000, Luca Boccassi wrote:
> > > > > > > > On Sun, 2021-03-14 at 22:10 +0000, Richard Purdie wrote:
> > > > > > > > > On Thu, 2021-03-11 at 15:09 +0000, luca.boccassi@gmail.com wrote:
> > > > > > > > > > From: Luca Boccassi <luca.boccassi@microsoft.com>
> > > > > > > > > > 
> > > > > > > > > > Recently util-linux gained an (optional) build dependency on libcryptsetup.
> > > > > > > > > > But libcryptsetup build-depends on util-linux for blkid (optional, can be disabled)
> > > > > > > > > > and uuid (mandatory).
> > > > > > > > > > Split out util-linux-uuid in a different recipe to break the cycle.
> > > > > > > > > > 
> > > > > > > > > > https://github.com/karelzak/util-linux/pull/898
> > > > > > > > > > 
> > > > > > > > > > Signed-off-by: Luca Boccassi <luca.boccassi@microsoft.com>
> > > > > > > > > 
> > > > > > > > > Unfortunately I noticed we had a performance regression in buildtimes in 
> > > > > > > > > recent changes. The closest I have this narrowed down to so far:
> > > > > > > > > 
> > > > > > > > > https://autobuilder.yocto.io/pub/non-release/20210314-14/testresults/buildperf-ubuntu1604/perf-ubuntu1604_master_20210314181831_d42487bf52.html
> > > > > > > > > 
> > > > > > > > > suggests it may be this change. I have more tests queued to confirm
> > > > > > > > > that definitively, if so we'll have to figure out why as this shouldn't
> > > > > > > > > really happen, its an 8% regression :(.
> > > > > > > > 
> > > > > > > > Very strange that a single recipe could do that - is there something
> > > > > > > > wrong in the new .bb that I missed and could cause this?
> > > > > > > 
> > > > > > > I'm wondering if it is because we're building util-linux twice now and
> > > > > > > there is some key choke point in the dependency chain. I have no evidence
> > > > > > > for that yet, it is just speculation though.
> > > > > > 
> > > > > > With the autoconf options I've set, on my laptop it takes 32s to do
> > > > > > configure + make -j2. Most of that is autoconf - make -j2 takes 8s.
> > > > > > 
> > > > > > Only 3 libraries are built with this combination: libcommon.a,
> > > > > > libtcolors.a, and libuuid.a/so. No executables or anything else is
> > > > > > built. It doesn't look like libtcolors is actually needed, I'll see if
> > > > > > I can prepare a patch to skip it, but I don't think it will buy more
> > > > > > than 1s, it's just two object files.
> > > > > > 
> > > > > > The good news is that meson support is about to land upstream, which
> > > > > > should be significantly faster than autoconf + make:
> > > > > > 
> > > > > > https://github.com/karelzak/util-linux/commits/topic/meson
> > > > > 
> > > > > Meson definitely improves the speed! I was wondering if it was from
> > > > > configure for example.
> > > > > 
> > > > > I now have more performance test results in (takes time to interleave 
> > > > > them with testing of master):
> > > > > 
> > > > > https://autobuilder.yocto.io/pub/non-release/20210315-1/testresults/buildperf-ubuntu1604/perf-ubuntu1604_master_20210315005048_6bb1621815.html
> > > > > 
> > > > > and I think this means it isn't from the util-linux change but one of 
> > > > > another three. I'm not entirely convinced those changes could do this
> > > > > but it is what the data says. 
> > > > > 
> > > > > I've queued more bisection to narrow it down from there...
> > > > 
> > > > BTW: this split also needs manual cleanup in the TMPDIR, right?
> > > 
> > > It shouldn't. The system should spot that util-linux has changed and uninstall
> > > it from the sysroots as it goes. There is something not working right there :(
> > 
> > I was wrong about that, the system doesn't have code for this, it has code for
> > the sysroots but not for other sstate tasks.
> > 
> > I think this is an oversight and we need to simplify things and make this cleanup
> > happen pre-build, much like the "unreachable" tasks cleanup happens today. If we
> > do make this happen, we probably need to add parallelism as the number of stale
> > sstate tasks being cleaned could be substantial.
> > 
> > Cheers,
> > 
> > Richard
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >