non-recursive build system

non-recursive build system

[Karel Zak]

 The original version was implemented by Diego Elio 'Flameeyes' Pettenò
 two years ago:
     http://thread.gmane.org/gmane.linux.utilities.util-linux-ng/3297

 See also:
     http://www.flameeyes.eu/autotools-mythbuster/automake/nonrecursive.html

 My current version is available at github, 'non-recursive' branch:
     https://github.com/karelzak/util-linux/commits/non-recursive


 Notes:

 * generate one Makefile for whole package

 * make(1) has all information to run tasks in parallel 

 * numbers (make -j)

             recursive | non-recursive
             ----------+--------------
    2 cores: 14.5 sec  | 13.2 sec
   16 cores:  9.5 sec  |  4.3 sec

   (... more CPU cores means more fun;-)

 * automake variables and rules are defined in subdir/Makemodule.am files
   (create one huge Makefile.am is too crazy idea)

 * top-level Makefile.am includes all Makemodule.am files

 * $(top_srcdir) prefix for lib/ and include/ is unnecessary, just use
   lib/bar.c and include/bar.h in makefiles

 * lib/ has been converted to libcommon.la, it means that
       foo_LDADD = libcommon.la
   is enough to use arbitrary stuff from lib/.

 * all rules are interpreted from top-level directory, it means that
       cd subdir; make
   or
       make -C subdir
   does not work.

 * binaries are stored in top-level (build) directory, you don't
   have to care about directories, just type:

	$ make mount
	$ ./mount --help

 * all build targets (including test binaries) are visible for bash completion
   scripts (e.g. /etc/bash_completion.d/make), so <tab> is your good friend


 * disadvantages:

    - full paths in makefiles (e.g. foo_SOURCE = subdir/foo.c)

    - we can't use the same name for subdir/ and for a final binary
      Necessary changes:

       - remove partx/ directory, move partx.c to disk-utils/
       - remove getopt/ directory, move getopt.c to misc-utils/
       - remove hwclock/ directory, move hwclock.c to sys-utils/
       - rename fdisk/ to fdisks/
       - rename mount/ to mount-deprecated/ (we use new sys-utils/mount.c)

      Note that consolidate the code and remove some subdirs (e.g. getopt/) is
      probably a good idea independently on the build-system changes.


 Comments & objections?

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: non-recursive build system

[Davidlohr Bueso]

On Tue, 2012-06-12 at 12:03 +0200, Karel Zak wrote:
> 
>  The original version was implemented by Diego Elio 'Flameeyes' Pettenò
>  two years ago:
>      http://thread.gmane.org/gmane.linux.utilities.util-linux-ng/3297
> 
>  See also:
>      http://www.flameeyes.eu/autotools-mythbuster/automake/nonrecursive.html
> 
>  My current version is available at github, 'non-recursive' branch:
>      https://github.com/karelzak/util-linux/commits/non-recursive
> 
> 
>  Notes:
> 
>  * generate one Makefile for whole package
> 
>  * make(1) has all information to run tasks in parallel 
> 
>  * numbers (make -j)
> 
>              recursive | non-recursive
>              ----------+--------------
>     2 cores: 14.5 sec  | 13.2 sec
>    16 cores:  9.5 sec  |  4.3 sec
> 
>    (... more CPU cores means more fun;-)

Nice, are these real cores or HT?

> 
>  * automake variables and rules are defined in subdir/Makemodule.am files
>    (create one huge Makefile.am is too crazy idea)
> 
>  * top-level Makefile.am includes all Makemodule.am files
> 
>  * $(top_srcdir) prefix for lib/ and include/ is unnecessary, just use
>    lib/bar.c and include/bar.h in makefiles
> 
>  * lib/ has been converted to libcommon.la, it means that
>        foo_LDADD = libcommon.la
>    is enough to use arbitrary stuff from lib/.
> 
>  * all rules are interpreted from top-level directory, it means that
>        cd subdir; make
>    or
>        make -C subdir
>    does not work.
> 
>  * binaries are stored in top-level (build) directory, you don't
>    have to care about directories, just type:
> 
> 	$ make mount
> 	$ ./mount --help
> 
>  * all build targets (including test binaries) are visible for bash completion
>    scripts (e.g. /etc/bash_completion.d/make), so <tab> is your good friend
> 
> 
>  * disadvantages:
> 
>     - full paths in makefiles (e.g. foo_SOURCE = subdir/foo.c)
> 
>     - we can't use the same name for subdir/ and for a final binary
>       Necessary changes:
>        - remove partx/ directory, move partx.c to disk-utils/
>        - remove getopt/ directory, move getopt.c to misc-utils/
>        - remove hwclock/ directory, move hwclock.c to sys-utils/
>        - rename fdisk/ to fdisks/

fdisk-utils?

>        - rename mount/ to mount-deprecated/ (we use new sys-utils/mount.c)
> 
>       Note that consolidate the code and remove some subdirs (e.g. getopt/) is
>       probably a good idea independently on the build-system changes.
> 

Thanks,
Davidlohr

Re: non-recursive build system

[Karel Zak]

On Tue, Jun 12, 2012 at 02:57:16PM +0200, Davidlohr Bueso wrote:
> On Tue, 2012-06-12 at 12:03 +0200, Karel Zak wrote:
> >  * numbers (make -j)
> > 
> >              recursive | non-recursive
> >              ----------+--------------
> >     2 cores: 14.5 sec  | 13.2 sec
> >    16 cores:  9.5 sec  |  4.3 sec
> > 
> >    (... more CPU cores means more fun;-)
> 
> Nice, are these real cores or HT?

 real cores, the machine with 16 cores was AMD from RH lab

 Unfortunately I have no such machine at home. Any HW sponsor around? :-))

> >        - rename fdisk/ to fdisks/
> 
> fdisk-utils?

 The ideal solution is to have libfdisk/ and then we can add small
 fdisk.c, sfdisk.c and cfdisk.c files to disk-utils/. (See mount.c and
 libmount/.)


-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: non-recursive build system

[Davidlohr Bueso]

On Tue, 2012-06-12 at 17:18 +0200, Karel Zak wrote:
> On Tue, Jun 12, 2012 at 02:57:16PM +0200, Davidlohr Bueso wrote:
> > On Tue, 2012-06-12 at 12:03 +0200, Karel Zak wrote:
> > >  * numbers (make -j)
> > > 
> > >              recursive | non-recursive
> > >              ----------+--------------
> > >     2 cores: 14.5 sec  | 13.2 sec
> > >    16 cores:  9.5 sec  |  4.3 sec
> > > 
> > >    (... more CPU cores means more fun;-)
> > 
> > Nice, are these real cores or HT?
> 
>  real cores, the machine with 16 cores was AMD from RH lab
> 
>  Unfortunately I have no such machine at home. Any HW sponsor around? :-))
> 
> > >        - rename fdisk/ to fdisks/
> > 
> > fdisk-utils?
> 
>  The ideal solution is to have libfdisk/ and then we can add small
>  fdisk.c, sfdisk.c and cfdisk.c files to disk-utils/. (See mount.c and
>  libmount/.)

well yes, that's where we're headed :)

> 
> 

Re: non-recursive build system

[Karel Zak]

On Tue, Jun 12, 2012 at 12:03:40PM +0200, Karel Zak wrote:
> 
> 
>  The original version was implemented by Diego Elio 'Flameeyes' Pettenò
>  two years ago:
>      http://thread.gmane.org/gmane.linux.utilities.util-linux-ng/3297
> 
>  See also:
>      http://www.flameeyes.eu/autotools-mythbuster/automake/nonrecursive.html
> 
>  My current version is available at github, 'non-recursive' branch:
>      https://github.com/karelzak/util-linux/commits/non-recursive
> 

 Merged into master branch, "git clean -xdf" after git pull is your
 good friend. Good luck ;-)

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com