util-linux and distro bootstrap, dependencies and build cycles

util-linux and distro bootstrap, dependencies and build cycles

[Stanislav Brabec]

Hallo.

Our distro build team has problems with the new util-linux. util-linux
is one of the low level packages. Now it depends on a two high level
packages with lots of dependencies:

systemd
python

Complete safe bootstrap distro rebuild requires two rebuilds of all
packages in the dependency cycle: one after initial util-linux rebuild
and second after high level dependency rebuild and consequent second
util-linux rebuild.

The build cycle util-linux <-> python is far the largest build cycle. In
case of forthcoming SLES12 it triggers rebuild of 2921 packages!

I was asked to solve this problem, and I created a solution that makes
possible to build:
util-linux stage 1: Everything except stuff that supports systemd and
  python-libmount
util-linux stage 2: Stuff that supports systemd (lslogins, logger, uuidd
  and fstrim service files) using installed low level stage 1 stuff.
python-libmount: Python bindings only using installed stage 1 stuff.

This is now implemented by a set of sed scripts on top of .am and .ac
sources. But I would like to create more generic solution.

What do you think about one of the following solutions?:

Solution 1:

Modify make system by adding:
--enable-build-stage-1: Distro bootstrap stage 1: Build only code
    without systemd and python dependencies.
--enable-build-stage-2: Distro bootstrap stage 2: Build code with
    systemd dependencies (even if they are optional). Use installed
    instances of libmount, libblkid, libsmartcols and libuuidd, do not
    build them in-tree.
--enable-build-stage-3: Distro bootstrap stage 3: Build python-libmount
    only. Use installed instance of libmount, do not build them in-tree.

Solution 2:

--enable-build-stage-1 and --enable-build-stage-2 exactly as above.

Split python-libmount into a separate project.

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                          e-mail: sbrabec@suse.cz
Lihovarská 1060/12                            tel: +49 911 7405384547
190 00 Praha 9                                 fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76

Re: util-linux and distro bootstrap, dependencies and build cycles

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 692 bytes --]

On Mon 11 Aug 2014 16:47:11 Stanislav Brabec wrote:
> Solution 1:
> 
> Modify make system by adding:
> --enable-build-stage-1: Distro bootstrap stage 1: Build only code
>     without systemd and python dependencies.
> --enable-build-stage-2: Distro bootstrap stage 2: Build code with
>     systemd dependencies (even if they are optional). Use installed
>     instances of libmount, libblkid, libsmartcols and libuuidd, do not
>     build them in-tree.
> --enable-build-stage-3: Distro bootstrap stage 3: Build python-libmount
>     only. Use installed instance of libmount, do not build them in-tree.

how is this any different from just passing --without-{udev,python} ?
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: util-linux and distro bootstrap, dependencies and build cycles

[Karel Zak]

On Mon, Aug 11, 2014 at 04:47:11PM +0200, Stanislav Brabec wrote:
> The build cycle util-linux <-> python is far the largest build cycle. In
> case of forthcoming SLES12 it triggers rebuild of 2921 packages!

well, util-linux is source tarball, all depends what do you want to
build from the tarball. You can rebuild and use only the binding and
ignore everything else.

  ./configure --disable-all-programs --enable-libblkid --enable-libuuid
              --enable-libmount --enable-pylibmount


I currently see only one disadvantage, we don't have "install-pylibmount" 
Makefile target now, you can use

  make install-pylibmountexecPYTHON install-pylibmountexecLTLIBRARIES

but it sucks..


Anyway, I don't understand why do you want to rebuild all recursively
(so result is rebuild of 2921 packages).

> I was asked to solve this problem, and I created a solution that makes
> possible to build:
> util-linux stage 1: Everything except stuff that supports systemd and
>   python-libmount
> util-linux stage 2: Stuff that supports systemd (lslogins, logger, uuidd
>   and fstrim service files) using installed low level stage 1 stuff.
> python-libmount: Python bindings only using installed stage 1 stuff.
> 
> This is now implemented by a set of sed scripts on top of .am and .ac
> sources. But I would like to create more generic solution.

Hmm.. modify build system is bad idea.

> What do you think about one of the following solutions?:
> 
> Solution 1:
> 
> Modify make system by adding:
> --enable-build-stage-1: Distro bootstrap stage 1: Build only code
>     without systemd and python dependencies.

 --without-systemd --without-python

> --enable-build-stage-2: Distro bootstrap stage 2: Build code with
>     systemd dependencies (even if they are optional). Use installed
>     instances of libmount, libblkid, libsmartcols and libuuidd, do not
>     build them in-tree.

 Why do you do not want to use in-tree code? The libraries interfaces 
 are versioned, it's safe to build against in-tree code and the
 install..

 We can add --with-outtree-libs or so, but it will require check for
 many things and it will increase complexity etc. IMHO it's really bad
 idea.

> --enable-build-stage-3: Distro bootstrap stage 3: Build python-libmount
>     only. Use installed instance of libmount, do not build them in-tree.

> Solution 2:
> 
> --enable-build-stage-1 and --enable-build-stage-2 exactly as above.
> 
> Split python-libmount into a separate project.


 Solution 3:

 Fix your build system to use complete build roots.
 
 You can use old version of your distro to build new complete
 util-linux (and another packages), then create new build root from
 the new set of the packages and rebuild again.


    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: util-linux and distro bootstrap, dependencies and build cycles

[Stanislav Brabec]

Mike Frysinger wrote:
> On Mon 11 Aug 2014 16:47:11 Stanislav Brabec wrote:
> > Solution 1:
> > 
> > Modify make system by adding:
> > --enable-build-stage-1: Distro bootstrap stage 1: Build only code
> >     without systemd and python dependencies.
> > --enable-build-stage-2: Distro bootstrap stage 2: Build code with
> >     systemd dependencies (even if they are optional). Use installed
> >     instances of libmount, libblkid, libsmartcols and libuuidd, do not
> >     build them in-tree.
> > --enable-build-stage-3: Distro bootstrap stage 3: Build python-libmount
> >     only. Use installed instance of libmount, do not build them in-tree.
> 
> how is this any different from just passing --without-{udev,python} ?

--without-systemd still builds the same set of binaries, just without
support of systemd. It does not allow to exclude lslogins, logger and
uuidd and build them later.

--with-python builds complete util-linux and libraries plus python
bindings, not bindings only.

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                          e-mail: sbrabec@suse.cz
Lihovarská 1060/12                            tel: +49 911 7405384547
190 00 Praha 9                                 fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76

Re: util-linux and distro bootstrap, dependencies and build cycles

[Stanislav Brabec]

Karel Zak wrote:
> On Mon, Aug 11, 2014 at 04:47:11PM +0200, Stanislav Brabec wrote:
> > The build cycle util-linux <-> python is far the largest build cycle. In
> > case of forthcoming SLES12 it triggers rebuild of 2921 packages!
> 
> well, util-linux is source tarball, all depends what do you want to
> build from the tarball. You can rebuild and use only the binding and
> ignore everything else.
> 
>   ./configure --disable-all-programs --enable-libblkid --enable-libuuid
>               --enable-libmount --enable-pylibmount
> 

This does not do what I need: It builds python bindings plus it builds
and installs libblkid, libmount and libuuid (for the second time, as we
already built and install them in the stage 1).

I need to build util-linux with defined configure options, but do it in
three stages with defined dependencies.

There is currently no way how to build python-libmount against
system-installed instance of libmount and not the in-tree one. (It means
that python-libmount forces to build libmount in-tree.)

> I currently see only one disadvantage, we don't have "install-pylibmount" 
> Makefile target now, you can use
> 
>   make install-pylibmountexecPYTHON install-pylibmountexecLTLIBRARIES
> but it sucks..

My proposal is different: Use special configuration options, that will
configure only part of util-linux to be built. Then use standard "make
install". It can be done by setting BUILD_* properly and using variable
instead of libxxx.la.

I think about using $(LIBMOUNT_LIBS) instead of libmount.la (and the
same for other in-tree libraries). LIBMOUNT_LIBS will be libmount.la
(use in-tree libmount) during normal build, and e. g. -lmount (use
installed libmount) during multi-stage build.

This will allow to build and install util-linux per-partes. All other
configure options could be kept intact.

> Anyway, I don't understand why do you want to rebuild all recursively
> (so result is rebuild of 2921 packages).

Imagine that util-linux is updated and automatic system evaluates
dependencies in a safe way:

1) util-linux update is built

2) all high level packages are triggered to rebuild by util-linux
   changes
   systemd and python are two of triggered packages

3) 2920 packages are rebuilt => all dependencies of systemd and python
   are ready => python is rebuilt, systemd was rebuild as well

4) python (and/or systemd) changes trigger util-linux to rebuild

5) all high level packages are triggered to rebuild by util-linux
   changes, including 2921 packages already rebuilt in 3)


How it looks with multi stage util-linux build:

1) util-linux stage 1 update is built

2) all high level packages are triggered to rebuild by util-linux
   changes
   systemd and python are two of triggered packages

3) all dependencies of systemd are ready => systemd is rebuilt => 
   util-linux-systemd (= util-linux stage 2) update is built

4) 2920 packages are rebuilt => python is rebuilt =>
   python-libmount (= util-linux stage 3) update is built

In case of openSUSE Build Service, multi-stage build saves one day of
machine time of our build cluster.

> > I was asked to solve this problem, and I created a solution that makes
> > possible to build:
> > util-linux stage 1: Everything except stuff that supports systemd and
> >   python-libmount
> > util-linux stage 2: Stuff that supports systemd (lslogins, logger, uuidd
> >   and fstrim service files) using installed low level stage 1 stuff.
> > python-libmount: Python bindings only using installed stage 1 stuff.
> > 
> > This is now implemented by a set of sed scripts on top of .am and .ac
> > sources. But I would like to create more generic solution.
> 
> Hmm.. modify build system is bad idea.

That is why I want to implement configure options and do it in a safe
and upstream-acceptable way.

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                          e-mail: sbrabec@suse.cz
Lihovarská 1060/12                            tel: +49 911 7405384547
190 00 Praha 9                                 fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76

Re: util-linux and distro bootstrap, dependencies and build cycles

[Mike Frysinger]

[-- Attachment #1: Type: text/plain, Size: 1284 bytes --]

On Tue 12 Aug 2014 22:15:31 Stanislav Brabec wrote:
> Mike Frysinger wrote:
> > On Mon 11 Aug 2014 16:47:11 Stanislav Brabec wrote:
> > > Solution 1:
> > > 
> > > Modify make system by adding:
> > > --enable-build-stage-1: Distro bootstrap stage 1: Build only code
> > >     without systemd and python dependencies.
> > > --enable-build-stage-2: Distro bootstrap stage 2: Build code with
> > >     systemd dependencies (even if they are optional). Use installed
> > >     instances of libmount, libblkid, libsmartcols and libuuidd, do not
> > >     build them in-tree.
> > > --enable-build-stage-3: Distro bootstrap stage 3: Build python-libmount
> > >     only. Use installed instance of libmount, do not build them in-tree.
> > 
> > how is this any different from just passing --without-{udev,python} ?
> 
> --without-systemd still builds the same set of binaries, just without
> support of systemd. It does not allow to exclude lslogins, logger and
> uuidd and build them later.
> 
> --with-python builds complete util-linux and libraries plus python
> bindings, not bindings only.

i'm not seeing the problem.  disable both, and you've broken the circular 
deps.  every day Gentoo is doing bootstraps of the distro and this isn't an 
issue for us.
-mike

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: util-linux and distro bootstrap, dependencies and build cycles

[Karel Zak]

On Tue, Aug 12, 2014 at 10:15:31PM +0200, Stanislav Brabec wrote:
> Mike Frysinger wrote:
> > On Mon 11 Aug 2014 16:47:11 Stanislav Brabec wrote:
> > > Solution 1:
> > > 
> > > Modify make system by adding:
> > > --enable-build-stage-1: Distro bootstrap stage 1: Build only code
> > >     without systemd and python dependencies.
> > > --enable-build-stage-2: Distro bootstrap stage 2: Build code with
> > >     systemd dependencies (even if they are optional). Use installed
> > >     instances of libmount, libblkid, libsmartcols and libuuidd, do not
> > >     build them in-tree.
> > > --enable-build-stage-3: Distro bootstrap stage 3: Build python-libmount
> > >     only. Use installed instance of libmount, do not build them in-tree.
> > 
> > how is this any different from just passing --without-{udev,python} ?
> 
> --without-systemd still builds the same set of binaries, just without
> support of systemd. It does not allow to exclude lslogins, logger and
> uuidd and build them later.

 then you need --disable-lslogins --disable-logger ..etc.

 If any --disable-* is missing (it's currently not defined for all
 utils) then we can improve configure.ac and the the new option.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: util-linux and distro bootstrap, dependencies and build cycles

[Karel Zak]

On Tue, Aug 12, 2014 at 10:15:36PM +0200, Stanislav Brabec wrote:
> Karel Zak wrote:
> > On Mon, Aug 11, 2014 at 04:47:11PM +0200, Stanislav Brabec wrote:
> > > The build cycle util-linux <-> python is far the largest build cycle. In
> > > case of forthcoming SLES12 it triggers rebuild of 2921 packages!
> > 
> > well, util-linux is source tarball, all depends what do you want to
> > build from the tarball. You can rebuild and use only the binding and
> > ignore everything else.
> > 
> >   ./configure --disable-all-programs --enable-libblkid --enable-libuuid
> >               --enable-libmount --enable-pylibmount
> > 
> 
> This does not do what I need: It builds python bindings plus it builds
> and installs libblkid, libmount and libuuid (for the second time, as we
> already built and install them in the stage 1).
> 
> I need to build util-linux with defined configure options, but do it in
> three stages with defined dependencies.

Sorry, but I don't plan to support and maintain any stages options,
all we need is to have --disable-<utilname> and then you can define
the stages in *your* build scripts (spec files).

> There is currently no way how to build python-libmount against
> system-installed instance of libmount and not the in-tree one. (It means
> that python-libmount forces to build libmount in-tree.)

Sure, nobody wants to support build against random out-of-tree libs
when we develop all together to make things consistent.

What is wrong with in-tree libs? After "make install" all the binaries 
is possible to use against out-of-tree libs. Problem solved.

(Well, we can improve "make install" to not install .so libs when 
 only python binding is wanted.)

> I think about using $(LIBMOUNT_LIBS) instead of libmount.la (and the
> same for other in-tree libraries). LIBMOUNT_LIBS will be libmount.la
> (use in-tree libmount) during normal build, and e. g. -lmount (use
> installed libmount) during multi-stage build.

You have to also guarantee that in-tree version is exactly the same
as -lmount version...

> > Anyway, I don't understand why do you want to rebuild all recursively
> > (so result is rebuild of 2921 packages).
> 
> Imagine that util-linux is updated and automatic system evaluates
> dependencies in a safe way

This is not answer to the question. I don't understand why Suse does
not use regular build roots for updates. For example for Fedora we
don't rebuild all from scratch -- just use the current system packages
to build new version (and then update the build machines). It's about
minutes to rebuild the package. 

Yes, I can imagine self-hosting distro, but it's IMHO very special
situation... (new CentOS from SRPMs, Linux from scratch, etc.), but
for standard updates?

> How it looks with multi stage util-linux build:
> 
> 1) util-linux stage 1 update is built
> 
> 2) all high level packages are triggered to rebuild by util-linux
>    changes
>    systemd and python are two of triggered packages
> 
> 3) all dependencies of systemd are ready => systemd is rebuilt => 
>    util-linux-systemd (= util-linux stage 2) update is built
> 
> 4) 2920 packages are rebuilt => python is rebuilt =>
>    python-libmount (= util-linux stage 3) update is built


Sure, all this is already possible (well, there is no --disable-logger, but 
we can add this option). All you need is to carefully specify --with-* and
--disable-*.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: util-linux and distro bootstrap, dependencies and build cycles

[Stanislav Brabec]

Karel Zak wrote:
>Sorry, but I don't plan to support and maintain any stages options,
>all we need is to have --disable-<utilname> and then you can define
>the stages in *your* build scripts (spec files).

> Sure, nobody wants to support build against random out-of-tree libs
> when we develop all together to make things consistent.

They are not random libraries. They are util-linux libraries built from
the same source code, just in time between there was python and systemd
rebuilt.

> What is wrong with in-tree libs? After "make install" all the binaries 
> is possible to use against out-of-tree libs. Problem solved.

There is nothing wrong with them. It is just a second build of the same
code, and not building them would save only few seconds of build. I
consider it a bit cleaner: always link against library instances that
will actually appear in the installed system.

However linking against an identical library with a different time stamp
than the installed one makes no problems in Linux, and no other
platforms are supported.

> (Well, we can improve "make install" to not install .so libs when 
>  only python binding is wanted.)

What about adding just --use-installed-libxxx? It would fit multi-stage
build needs as well and it would be cleaner to implement. I think that
not-installing built shared libraries is not supported by autotools
(IMHO noinst switches to static linking).

To be sure about installed version, it's easy to add pkg-config version
check.
PKG_CHECK_MODULES([LIBMOUNT], [mount = $PACKAGE_VERSION],...)

If you would accept such change, I will send a patch.

> You have to also guarantee that in-tree version is exactly the same
> as -lmount version...

It is guaranteed by the build system.

> > > Anyway, I don't understand why do you want to rebuild all recursively
> > > (so result is rebuild of 2921 packages).
> > 
> > Imagine that util-linux is updated and automatic system evaluates
> > dependencies in a safe way
> 
> This is not answer to the question. I don't understand why Suse does
> not use regular build roots for updates. For example for Fedora we
> don't rebuild all from scratch -- just use the current system packages
> to build new version (and then update the build machines). It's about
> minutes to rebuild the package.

SUSE does not rebuild all from scratch for standard packages.

openSUSE Build Service is a multi-repository system. Any change in low
level package will trigger all higher level packages in connected
repositories to rebuild, so the Build Service guarantees, that all high
level packages are build against the correct version of the low level
package.

If the package rebuild triggers change, the rebuild trigger is activated
as well.

util-linux is a part of the base build image, so its change (correctly)
triggers all packages to rebuild.

If python-libmount is part of the util-linux package, any change in
python package or its dependencies would trigger util-linux to rebuild,
and the change in the binary image of python binding would trigger whole
distro to rebuild.

> 
> Yes, I can imagine self-hosting distro, but it's IMHO very special
> situation... (new CentOS from SRPMs, Linux from scratch, etc.), but
> for standard updates?

Only base system changes (gcc, binutils, bash, util-linux,...) trigger
whole distro to rebuild. Updates of high level packages triggers only
dependent package.

This prevents any binary incompatibility when releasing rolling update
with a new library.

This also makes possible to create experimental repositories with single
library updated. The build system automatically identifies and rebuilds
its dependencies.

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                          e-mail: sbrabec@suse.cz
Lihovarská 1060/12                            tel: +49 911 7405384547
190 00 Praha 9                                 fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76

Re: util-linux and distro bootstrap, dependencies and build cycles

[Linda Walsh]

Stanislav Brabec wrote:
> Only base system changes (gcc, binutils, bash, util-linux,...) trigger
> whole distro to rebuild. Updates of high level packages triggers only
> dependent package.
>   
> This prevents any binary incompatibility when releasing rolling update
> with a new library.
>   
----
There are other ways of dealing with binary compatibility.

As it is, if you want to update perl, python, ruby  for example, you need
to rebuild the entire system.

Even vim/Gvim has dependencies on systemd because it supplies config files
that systemd uses.  But worse, if you update perl/python or ruby, you can
no longer edit. 

    It has been pointed to suse, multiple times, that they could build 
most of their
utils with less hard-coded dependencies (like have gvim dynamically load 
script
engines at run time on demand, when needed rather than having to load 
all of them
at load time in order for gvim to run.


    Even when no binary dependencies are  present, hard-coding versions 
and labels into
each of the binaries to only work with recognized components guarantees 
failure on
opensuse systems.  I have pushed for having a more robust system, but 
the current
mantra is "break everything" if the user changes anything ....

Re: util-linux and distro bootstrap, dependencies and build cycles

[Karel Zak]

On Wed, Aug 13, 2014 at 04:53:40PM +0200, Stanislav Brabec wrote:
> > What is wrong with in-tree libs? After "make install" all the binaries 
> > is possible to use against out-of-tree libs. Problem solved.
> 
> There is nothing wrong with them. It is just a second build of the same
> code, and not building them would save only few seconds of build. I

 Hmm.. premature optimization? :-)

> consider it a bit cleaner: always link against library instances that
> will actually appear in the installed system.
> 
> However linking against an identical library with a different time stamp
> than the installed one makes no problems in Linux, and no other
> platforms are supported.
> 
> > (Well, we can improve "make install" to not install .so libs when 
> >  only python binding is wanted.)
> 
> What about adding just --use-installed-libxxx? It would fit multi-stage

 Now, when I think about it.. it seem the extra option is unnecessary

   --disable-libmount --enable-pylibmount

 provides all necessary information. If the Py binding is explicitly
 required, but the library is disabled then it's obvious that
 out-of-tree libs are necessary. Now we use

   UL_REQUIRES_BUILD([pylibmount], [libmount])

 to define dependence between in-tree builds, maybe it would be
 possible to add a new macro:

   UL_REQUIRES_BUILD_OR_LIB([pylibmount], [libmount], [MOUNT])
  
 where 'MOUNT' will be used for

   PKG_CHECK_MODULES([MOUNT])
 
 if $build_libmount=no, see m4/ul.m4 for more details.


 But again, it will increase configure complexity, IMHO it would be 
 enough to improve "make install" to install only the Py binding and
 always build against in-tree libs. If your distro build system
 triggers so many rebuilds then save few seconds is a real detail :-)

> SUSE does not rebuild all from scratch for standard packages.
> 
> openSUSE Build Service is a multi-repository system. Any change in low
> level package will trigger all higher level packages in connected
> repositories to rebuild, so the Build Service guarantees, that all high
> level packages are build against the correct version of the low level
> package.

 You will see more and more problems with this concept. The border
 between high-level and low-level is fragile and difficult to define.
 The typical problem are interpreted languages or systemd.

> util-linux is a part of the base build image, so its change (correctly)
> triggers all packages to rebuild.

 but why all packages? Why not packages which depend on util-linux libs
 only?

> If python-libmount is part of the util-linux package, any change in
> python package or its dependencies would trigger util-linux to rebuild,
> and the change in the binary image of python binding would trigger whole
> distro to rebuild.

 OK, this makes sense.

> Only base system changes (gcc, binutils, bash, util-linux,...) trigger
> whole distro to rebuild. Updates of high level packages triggers only
> dependent package.

 hmm.. we (fedora/rhel) also do mass rebuilds, but it's always
 triggered manually and usually for some obvious issue which affect
 all packages (gcc/libc bugs etc.).

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: util-linux and distro bootstrap, dependencies and build cycles

[Stanislav Brabec]

Karel Zak wrote:
> On Wed, Aug 13, 2014 at 04:53:40PM +0200, Stanislav Brabec wrote:
>
>  Now, when I think about it.. it seem the extra option is unnecessary
> 
>    --disable-libmount --enable-pylibmount
> 
>  provides all necessary information. If the Py binding is explicitly
>  required, but the library is disabled then it's obvious that
>  out-of-tree libs are necessary. Now we use
> 
>    UL_REQUIRES_BUILD([pylibmount], [libmount])
> 
>  to define dependence between in-tree builds, maybe it would be
>  possible to add a new macro:
> 
>    UL_REQUIRES_BUILD_OR_LIB([pylibmount], [libmount], [MOUNT])
>   
>  where 'MOUNT' will be used for
> 
>    PKG_CHECK_MODULES([MOUNT])

Yes, it could work. I'll try to create a patch.

libmount is not wanted + pylibmount is wanted
  => check for external libmount

> > util-linux is a part of the base build image, so its change (correctly)
> > triggers all packages to rebuild.
> 
>  but why all packages? Why not packages which depend on util-linux libs
>  only?

util-linux is part of the base build image together with gcc binutils,
glibc, sed and few other packages.
=> Any package can be affected by util-linux change

> > Only base system changes (gcc, binutils, bash, util-linux,...) trigger
> > whole distro to rebuild. Updates of high level packages triggers only
> > dependent package.
> 
>  hmm.. we (fedora/rhel) also do mass rebuilds, but it's always
>  triggered manually and usually for some obvious issue which affect
>  all packages (gcc/libc bugs etc.).

I know. It may easily cause overseeing of newly introduced fulfilling:
PKG_CHECK_MODULES([foo >= version])

SUSE does no manual rebuilds. It depends on automatic rebuild triggers.
Mass rebuilds happen automatically if somebody submits a package in the
core build image. In all other cases, only part of the tree is
triggered.

There are several ways, how to automatically trigger rebuilds:
- Trigger by any change in lower level packages.
- Trigger by change in provided symbols (file list, version, SONAME,
  library symbols).
- Trigger by change apparently that apparently caused breakage (SONAME
  change).

openSUSE Build Service uses first two triggers. For example, Gentoo uses
the third way.

The first two triggers have to solve dependency loops. They introduce
multiple triggering of rebuilds of the same package:
new util-linux => rebuild python (it can change)
new python => rebuild util-linux (as python-libmount will change)
util-linux changed (exactly python-libmount changed) => rebuild python

In most cases, package contents settles after two build cycles (If the
resulting package is bit-equal to the previous one, rebuild trigger is
not activated.)

Large dependency loops are not wanted, as they affect distro build time.

Note: Far the worst dependency cycle happens between maven (Java build
system) and about 500 Java package that require maven to build, but they
are required by maven to build maven. In this case, our package
maintainer broken the dependency loop by using of binary images of
dependent packages.

-- 
Best Regards / S pozdravem,

Stanislav Brabec
software developer
---------------------------------------------------------------------
SUSE LINUX, s. r. o.                          e-mail: sbrabec@suse.cz
Lihovarská 1060/12                            tel: +49 911 7405384547
190 00 Praha 9                                 fax:  +420 284 084 001
Czech Republic                                    http://www.suse.cz/
PGP: 830B 40D5 9E05 35D8 5E27 6FA3 717C 209F A04F CD76