[PATCH 0/3] Suppress Texinfo utilities invoked at build time for -cross & -native recipes

[PATCH 0/3] Suppress Texinfo utilities invoked at build time for -cross & -native recipes

[Max Eliaser]

Hello list,

As per discussion with Richard Purdie and others, I'm submitting a patch to 
suppress some unnecessary invocations of utilities from the Texinfo suite during
Yocto builds. Specifically, it targets -native and -cross recipes that inherit
texinfo.bbclass.

The Texinfo utilities are suppressed by supplying doppelgänger executables which
understand the same command-line options as the genuine Texinfo utilites, and
which can create blank output files where expected, but which do no actual
processing, formatting, or markup. To see how this is done, look in the 
texinfo-dummy-native recipe (under meta/recipes-extended,) added in this patch
series, as well as the modifications to texinfo.bbclass, also in this patch 
series.

Suppressing the Texinfo utilities for -native and -cross recipes does provide a
noticable performance gain, as documented in bug 2421. [1] You could make the 
case that this patch series still does not completely address bug 2421, as there
are other utilities not from the Texinfo suite (groff, pod2man, etc) which are
not suppressed by these changes.

Target-architecture recipes continue to use the host system's Texinfo utilities
as before, so that dependency is not eliminated by this patch series. 

Regards,
-Max Eliaser

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=2421
The following changes since commit 7b78c364b7de901f97cd8dbe56d5e02f545bf3d3:

  cleanup-workdir: Fix crosssdk reference after addition of PN (2014-05-05 09:05:42 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib meliaser/texinfo-build
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=meliaser/texinfo-build

Max Eliaser (3):
  Allow texinfo, texinfo-native recipes to build w/o host system's
    Texinfo utils
  Add texinfo-dummy-native recipe w/ scripts to stand in for Texinfo
    utils.
  native/cross recipes use texinfo-dummy-native; target arch use host's
    Texinfo.

 meta/classes/texinfo.bbclass                       |  15 ++-
 meta/conf/bitbake.conf                             |   1 +
 .../texinfo-dummy-native/texinfo-dummy-native.bb   |  30 +++++
 .../texinfo-dummy-native/texinfo-dummy/COPYING     |  19 ++++
 .../texinfo-dummy-native/texinfo-dummy/template.py | 122 +++++++++++++++++++++
 .../texinfo/texinfo/disable-native-tools.patch     |  43 --------
 .../texinfo/texinfo/use_host_makedoc.patch         |  17 ---
 meta/recipes-extended/texinfo/texinfo_5.2.bb       |   8 +-
 8 files changed, 188 insertions(+), 67 deletions(-)
 create mode 100644 meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb
 create mode 100644 meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/COPYING
 create mode 100644 meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/template.py
 delete mode 100644 meta/recipes-extended/texinfo/texinfo/disable-native-tools.patch
 delete mode 100644 meta/recipes-extended/texinfo/texinfo/use_host_makedoc.patch

-- 
1.8.3.2

[PATCH 1/3] Allow texinfo, texinfo-native recipes to build w/o host system's Texinfo utils

[Max Eliaser]

This is necessary for the eventual removal of Yocto's current dependency on
the host system's Texinfo utilities.

The Texinfo suite's build system actually already has a self-bootstrap system
it uses when cross-compiling or when Texinfo binaries aren't already
available. This had been suppressed with patches, but this commit removes
those patches and alters the recipe to make the bootstrapping work properly.

This adds no real overhead for the texinfo-native recipe, but it does mean
that the target architecture recipe actually builds some native binaries
before building the cross-compiled ones. I find this extra overhead
acceptable, since I doubt too many people using Yocto are shipping
images with cross-compiled versions of the Texinfo utilities.

The removal of these patches also allows us to make the target arch texinfo
recipe no longer depend on texinfo-native, since we're going back to using the
built-in bootstrapping process as the Texinfo upstream maintainers intended.

Signed-off-by: Max Eliaser <max.eliaser@intel.com>
---
 .../texinfo/texinfo/disable-native-tools.patch     | 43 ----------------------
 .../texinfo/texinfo/use_host_makedoc.patch         | 17 ---------
 meta/recipes-extended/texinfo/texinfo_5.2.bb       |  8 +---
 3 files changed, 2 insertions(+), 66 deletions(-)
 delete mode 100644 meta/recipes-extended/texinfo/texinfo/disable-native-tools.patch
 delete mode 100644 meta/recipes-extended/texinfo/texinfo/use_host_makedoc.patch

diff --git a/meta/recipes-extended/texinfo/texinfo/disable-native-tools.patch b/meta/recipes-extended/texinfo/texinfo/disable-native-tools.patch
deleted file mode 100644
index ab6f165..0000000
--- a/meta/recipes-extended/texinfo/texinfo/disable-native-tools.patch
+++ /dev/null
@@ -1,43 +0,0 @@
-We already DEPEND on the native texinfo being present before building so
-there isn't any need to try and build the required native texinfo binaries
-before cross-compiling. This simplifies the recipe somewhat!
-
-Upstream-Status: Inappropriate oe specific
-
-Signed-off-by: Joshua Lock <josh@linux.intel.com>
-
-Index: texinfo-4.13/configure.ac
-===================================================================
---- texinfo-4.13.orig/configure.ac
-+++ texinfo-4.13/configure.ac
-@@ -100,29 +100,7 @@ AC_CANONICAL_BUILD
- # $native_tools is also added to SUBDIRS in the main Makefile.am,
- # so that make compiles the native tools first.
- #
--if test "$cross_compiling" = no; then
--  native_tools=
--else
--  native_tools=tools
--  test -d "$native_tools" || mkdir "$native_tools"
--  confdir=`(cd "$srcdir";pwd)`
--  # Make sure the secondary configure won't fail with
--  # "error: source directory already configured".
--  rm -f config.status
--  AC_MSG_NOTICE([[Doing configure of native tools (${build}).]])
--  cd "$native_tools" || exit 1
--  # Run secondary configure in alternate environment or
--  # it gets the wrong CC etc.
--  # env -i gives this build host configure a clean environment;
--  # consequently, we have to re-initialize $PATH.
--  env -i CC="$BUILD_CC" AR="$BUILD_AR" RANLIB="$BUILD_RANLIB" \
--      PATH="$PATH" \
--      tools_only=1 \
--    ${confdir}/configure --build=${build} --host=${build} \
--                         --disable-rpath --disable-nls
--  cd .. || exit 1
--  AC_MSG_NOTICE([[Continuing with main configure (${host}).]])
--fi
-+native_tools=
- AC_SUBST(native_tools)
- AM_CONDITIONAL(TOOLS_ONLY, [[test "x$tools_only" = x1]])
- 
diff --git a/meta/recipes-extended/texinfo/texinfo/use_host_makedoc.patch b/meta/recipes-extended/texinfo/texinfo/use_host_makedoc.patch
deleted file mode 100644
index 5b7f32d..0000000
--- a/meta/recipes-extended/texinfo/texinfo/use_host_makedoc.patch
+++ /dev/null
@@ -1,17 +0,0 @@
-Upstream-Status: Inappropriate [cross build specific]
-
-Signed-off-by: Saul Wold <sgw@linux.intel.com>
-
-Index: texinfo-5.1/info/Makefile.am
-===================================================================
---- texinfo-5.1.orig/info/Makefile.am
-+++ texinfo-5.1/info/Makefile.am
-@@ -76,7 +76,7 @@ cmd_sources = $(srcdir)/session.c $(srcd
- # more than once.
- funs.h: makedoc$(EXEEXT) $(cmd_sources)
- 	rm -f $(generated_sources)
--	$(top_builddir)/$(native_tools)/info/makedoc $(cmd_sources)
-+	makedoc $(cmd_sources)
- 
- # The following hack is necessary to hint make before the automatic
- # dependencies are built.
diff --git a/meta/recipes-extended/texinfo/texinfo_5.2.bb b/meta/recipes-extended/texinfo/texinfo_5.2.bb
index c1ccaa7..53e7e4f 100644
--- a/meta/recipes-extended/texinfo/texinfo_5.2.bb
+++ b/meta/recipes-extended/texinfo/texinfo_5.2.bb
@@ -7,19 +7,14 @@ SECTION = "console/utils"
 LICENSE = "GPLv3+"
 LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
 
-DEPENDS = "zlib ncurses texinfo-native"
+DEPENDS = "zlib ncurses"
 DEPENDS_class-native = "zlib-native ncurses-native"
 
-TARGET_PATCH = "file://use_host_makedoc.patch"
-TARGET_PATCH_class-native = ""
-
 SRC_URI = "${GNU_MIRROR}/texinfo/${BP}.tar.gz \
            file://texinfo-4.12-zlib.patch \
            file://texinfo-4.13a-powerpc.patch \
-           file://disable-native-tools.patch \
            file://link-zip.patch \
            file://dont-depend-on-help2man.patch \
-           ${TARGET_PATCH} \
           "
 
 SRC_URI[md5sum] = "1b8f98b80a8e6c50422125e07522e8db"
@@ -33,6 +28,7 @@ inherit gettext autotools
 do_compile_prepend() {
 	if [ -d tools ];then
 		oe_runmake -C tools/gnulib/lib
+		oe_runmake -C tools/info/ makedoc # TODO: Investigate whether this is an upstream bug
 	fi
 }
 
-- 
1.8.3.2

[PATCH 2/3] Add texinfo-dummy-native recipe w/ scripts to stand in for Texinfo utils.

[Max Eliaser]

More work toward eliminating the dependency on the host system's Texinfo--
Python scripts that understand the same command-line options as the Texinfo
utilities, and create blank output files if appropriate, but don't actually
do any of the work done by those utilities.

This will be necessary to avoid circular dependencies when we start explicitly
tracking dependencies on texinfo-native; i.e. texinfo-native ->
autoconf-native -> texinfo-native. If we have all native recipes that inherit
texinfo.bbclass depend on texinfo-dummy-native instead of texinfo-native, the
cycle is broken.

It may also provide a performance gain by skipping the actual work of
formatting and generating documentation files.

Signed-off-by: Max Eliaser <max.eliaser@intel.com>
---
 .../texinfo-dummy-native/texinfo-dummy-native.bb   |  30 +++++
 .../texinfo-dummy-native/texinfo-dummy/COPYING     |  19 ++++
 .../texinfo-dummy-native/texinfo-dummy/template.py | 122 +++++++++++++++++++++
 3 files changed, 171 insertions(+)
 create mode 100644 meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb
 create mode 100644 meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/COPYING
 create mode 100644 meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/template.py

diff --git a/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb b/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb
new file mode 100644
index 0000000..b5420a3
--- /dev/null
+++ b/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb
@@ -0,0 +1,30 @@
+DESCRIPTION = "Fake version of the texinfo utility suite"
+SECTION = "console/utils"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://COPYING;md5=d6bb62e73ca8b901d3f2e9d71542f4bb"
+DEPENDS = ""
+PV = "1.0"
+
+SRC_URI = "file://template.py file://COPYING"
+
+S = "${WORKDIR}"
+
+NATIVE_PACKAGE_PATH_SUFFIX = "/${PN}"
+
+inherit native
+
+do_install_name() {
+    FILENAME="${D}${bindir}/$1"
+    # Using ln causes problems with rm_work
+    cp -T "${S}/template.py" "$FILENAME"
+    chmod +x $FILENAME
+}
+
+do_install() {
+    mkdir -p "${D}${bindir}"
+    for i in makeinfo pod2texi texi2dvi pdftexi2dvi texindex texi2pdf \
+             txixml2texi texi2any install-info ginstall-info \
+             update-info-dir; do
+        do_install_name $i
+    done
+}
diff --git a/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/COPYING b/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/COPYING
new file mode 100644
index 0000000..531cbe7
--- /dev/null
+++ b/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/COPYING
@@ -0,0 +1,19 @@
+Copyright (c) 2014 Intel Corp.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
diff --git a/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/template.py b/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/template.py
new file mode 100644
index 0000000..4dc31a9
--- /dev/null
+++ b/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/template.py
@@ -0,0 +1,122 @@
+#! /usr/bin/env python2.7
+
+# template.py (and other filenames)
+# By Max Eliaser (max.eliaser@intel.com)
+
+# Copyright (c) 2014 Intel Corp.
+
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+
+# This program acts like a dummy version of the texinfo utilities, creating
+# the right output files but leaving them blank. It will parse out the name
+# of the executable from argv[0] and emulate the corresponding program, so
+# multiple copies of this script will exist under different names.
+
+import sys, os
+
+olong = "--output="
+Elong = "--macro-expand="
+
+
+this_binary = sys.argv[0].split ("/")[-1]
+
+# To be outputted if functionality that hasn't been stubbed yet is invoked.
+stub_msg = """
+This stand-in version of %s is not yet fully capable of emulating the real
+version from the GNU texinfo suite. If you see this message, file a bug report
+with details on the recipe that failed.
+""" % this_binary
+
+# Autotools setups query the version, so this is actually necessary. Some of
+# them (lookin' at you, eglibc) actually look for the substring "GNU texinfo,"
+# so we put that substring in there without actually telling a lie.
+version_str = """ %s (fake texinfo, emulating GNU texinfo) 5.2
+ 
+Super amazing version which is totally not fake in any way whatsoever.
+Copyright (C) 2014 Intel Corp. Distributed under the terms of the MIT
+license.
+""" % this_binary
+
+simple_binaries = "pod2texi texi2dvi pdftexi2dvi texindex texi2pdf \
+                   txixml2texi install-info ginstall-info \
+                   update-info-dir".split ()
+
+# These utilities use a slightly different set of options and flags.
+complex_binaries = "makeinfo texi2any".split ()
+
+valid_binaries = simple_binaries + complex_binaries
+
+# For generating blank output files.
+def touch_file (path):
+    f = open (path, "w")
+    f.close ()
+
+assert this_binary in valid_binaries, \
+       this_binary + " is not one of " + ', '.join (valid_binaries)
+
+if "--version" in sys.argv:
+    print version_str
+    sys.exit (0)
+
+# For debugging
+log_interceptions = False
+if log_interceptions:
+    f = open ("/tmp/intercepted_" + this_binary, "a")
+    f.write (' '.join ([this_binary] + sys.argv[1:]) + '\n')
+    f.close ()
+
+# Look through the options and flags, and if necessary, touch any output
+# files.
+arg_idx = 1
+while arg_idx < len (sys.argv):
+    arg = sys.argv [arg_idx]
+    
+    if arg == "--":
+        break
+    
+    # Something like -I . can result in a need for this (specifically the .)
+    elif len (arg) < 2:
+        pass
+    
+    # Check if -o or --output is specified. These can be used at most once.
+    elif arg[0] == '-' and arg[1] != '-' and arg[len (arg) - 1] == 'o':
+        touch_file (sys.argv[arg_idx + 1])
+        sys.exit (0)
+    elif arg.startswith (olong):
+        touch_file (arg.split ("=")[1])
+        sys.exit (0)
+    
+    # Check for functionality that isn't implemented yet.
+    else:
+        assert arg[1] == '-' or 'E' not in arg or \
+               this_binary in simple_binaries, \
+               "-E option not yet supported" + stub_msg
+        
+        assert not arg.startswith (Elong), \
+               Elong[:-1] + " option not yet supported" + stub_msg
+    
+    arg_idx += 1
+
+# The -o/--output option overrides the default. For makeinfo and texi2any,
+# that default is to look for a @setfilename command in the input file.
+# Otherwise, printing nothing to stdout and then exiting should suffice.
+assert this_binary in simple_binaries, \
+       "Don't know how to get default output file name from input file!" + \
+       stub_msg
+
-- 
1.8.3.2

Re: [PATCH 2/3] Add texinfo-dummy-native recipe w/ scripts to stand in for Texinfo utils.

[Saul Wold]

On 05/14/2014 02:36 PM, Max Eliaser wrote:
> More work toward eliminating the dependency on the host system's Texinfo--
> Python scripts that understand the same command-line options as the Texinfo
> utilities, and create blank output files if appropriate, but don't actually
> do any of the work done by those utilities.
>
> This will be necessary to avoid circular dependencies when we start explicitly
> tracking dependencies on texinfo-native; i.e. texinfo-native ->
> autoconf-native -> texinfo-native. If we have all native recipes that inherit
> texinfo.bbclass depend on texinfo-dummy-native instead of texinfo-native, the
> cycle is broken.
>
> It may also provide a performance gain by skipping the actual work of
> formatting and generating documentation files.
>
> Signed-off-by: Max Eliaser <max.eliaser@intel.com>
> ---
>   .../texinfo-dummy-native/texinfo-dummy-native.bb   |  30 +++++
>   .../texinfo-dummy-native/texinfo-dummy/COPYING     |  19 ++++
>   .../texinfo-dummy-native/texinfo-dummy/template.py | 122 +++++++++++++++++++++
>   3 files changed, 171 insertions(+)
>   create mode 100644 meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb
>   create mode 100644 meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/COPYING
>   create mode 100644 meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/template.py
>
I think these should live in texinfo instead of adding a new 
texinfo-dummp-native directory?

Sau!

> diff --git a/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb b/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb
> new file mode 100644
> index 0000000..b5420a3
> --- /dev/null
> +++ b/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb
> @@ -0,0 +1,30 @@
> +DESCRIPTION = "Fake version of the texinfo utility suite"
> +SECTION = "console/utils"
> +LICENSE = "MIT"
> +LIC_FILES_CHKSUM = "file://COPYING;md5=d6bb62e73ca8b901d3f2e9d71542f4bb"
> +DEPENDS = ""
> +PV = "1.0"
> +
> +SRC_URI = "file://template.py file://COPYING"
> +
> +S = "${WORKDIR}"
> +
> +NATIVE_PACKAGE_PATH_SUFFIX = "/${PN}"
> +
> +inherit native
> +
> +do_install_name() {
> +    FILENAME="${D}${bindir}/$1"
> +    # Using ln causes problems with rm_work
> +    cp -T "${S}/template.py" "$FILENAME"
> +    chmod +x $FILENAME
> +}
> +
> +do_install() {
> +    mkdir -p "${D}${bindir}"
> +    for i in makeinfo pod2texi texi2dvi pdftexi2dvi texindex texi2pdf \
> +             txixml2texi texi2any install-info ginstall-info \
> +             update-info-dir; do
> +        do_install_name $i
> +    done
> +}
> diff --git a/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/COPYING b/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/COPYING
> new file mode 100644
> index 0000000..531cbe7
> --- /dev/null
> +++ b/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/COPYING
> @@ -0,0 +1,19 @@
> +Copyright (c) 2014 Intel Corp.
> +
> +Permission is hereby granted, free of charge, to any person obtaining a copy
> +of this software and associated documentation files (the "Software"), to deal
> +in the Software without restriction, including without limitation the rights
> +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
> +copies of the Software, and to permit persons to whom the Software is
> +furnished to do so, subject to the following conditions:
> +
> +The above copyright notice and this permission notice shall be included in
> +all copies or substantial portions of the Software.
> +
> +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
> +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
> +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
> +THE SOFTWARE.
> diff --git a/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/template.py b/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/template.py
> new file mode 100644
> index 0000000..4dc31a9
> --- /dev/null
> +++ b/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/template.py
> @@ -0,0 +1,122 @@
> +#! /usr/bin/env python2.7
> +
> +# template.py (and other filenames)
> +# By Max Eliaser (max.eliaser@intel.com)
> +
> +# Copyright (c) 2014 Intel Corp.
> +
> +# Permission is hereby granted, free of charge, to any person obtaining a copy
> +# of this software and associated documentation files (the "Software"), to deal
> +# in the Software without restriction, including without limitation the rights
> +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
> +# copies of the Software, and to permit persons to whom the Software is
> +# furnished to do so, subject to the following conditions:
> +
> +# The above copyright notice and this permission notice shall be included in
> +# all copies or substantial portions of the Software.
> +
> +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
> +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
> +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
> +# THE SOFTWARE.
> +
> +# This program acts like a dummy version of the texinfo utilities, creating
> +# the right output files but leaving them blank. It will parse out the name
> +# of the executable from argv[0] and emulate the corresponding program, so
> +# multiple copies of this script will exist under different names.
> +
> +import sys, os
> +
> +olong = "--output="
> +Elong = "--macro-expand="
> +
> +
> +this_binary = sys.argv[0].split ("/")[-1]
> +
> +# To be outputted if functionality that hasn't been stubbed yet is invoked.
> +stub_msg = """
> +This stand-in version of %s is not yet fully capable of emulating the real
> +version from the GNU texinfo suite. If you see this message, file a bug report
> +with details on the recipe that failed.
> +""" % this_binary
> +
> +# Autotools setups query the version, so this is actually necessary. Some of
> +# them (lookin' at you, eglibc) actually look for the substring "GNU texinfo,"
> +# so we put that substring in there without actually telling a lie.
> +version_str = """ %s (fake texinfo, emulating GNU texinfo) 5.2
> +
> +Super amazing version which is totally not fake in any way whatsoever.
> +Copyright (C) 2014 Intel Corp. Distributed under the terms of the MIT
> +license.
> +""" % this_binary
> +
> +simple_binaries = "pod2texi texi2dvi pdftexi2dvi texindex texi2pdf \
> +                   txixml2texi install-info ginstall-info \
> +                   update-info-dir".split ()
> +
> +# These utilities use a slightly different set of options and flags.
> +complex_binaries = "makeinfo texi2any".split ()
> +
> +valid_binaries = simple_binaries + complex_binaries
> +
> +# For generating blank output files.
> +def touch_file (path):
> +    f = open (path, "w")
> +    f.close ()
> +
> +assert this_binary in valid_binaries, \
> +       this_binary + " is not one of " + ', '.join (valid_binaries)
> +
> +if "--version" in sys.argv:
> +    print version_str
> +    sys.exit (0)
> +
> +# For debugging
> +log_interceptions = False
> +if log_interceptions:
> +    f = open ("/tmp/intercepted_" + this_binary, "a")
> +    f.write (' '.join ([this_binary] + sys.argv[1:]) + '\n')
> +    f.close ()
> +
> +# Look through the options and flags, and if necessary, touch any output
> +# files.
> +arg_idx = 1
> +while arg_idx < len (sys.argv):
> +    arg = sys.argv [arg_idx]
> +
> +    if arg == "--":
> +        break
> +
> +    # Something like -I . can result in a need for this (specifically the .)
> +    elif len (arg) < 2:
> +        pass
> +
> +    # Check if -o or --output is specified. These can be used at most once.
> +    elif arg[0] == '-' and arg[1] != '-' and arg[len (arg) - 1] == 'o':
> +        touch_file (sys.argv[arg_idx + 1])
> +        sys.exit (0)
> +    elif arg.startswith (olong):
> +        touch_file (arg.split ("=")[1])
> +        sys.exit (0)
> +
> +    # Check for functionality that isn't implemented yet.
> +    else:
> +        assert arg[1] == '-' or 'E' not in arg or \
> +               this_binary in simple_binaries, \
> +               "-E option not yet supported" + stub_msg
> +
> +        assert not arg.startswith (Elong), \
> +               Elong[:-1] + " option not yet supported" + stub_msg
> +
> +    arg_idx += 1
> +
> +# The -o/--output option overrides the default. For makeinfo and texi2any,
> +# that default is to look for a @setfilename command in the input file.
> +# Otherwise, printing nothing to stdout and then exiting should suffice.
> +assert this_binary in simple_binaries, \
> +       "Don't know how to get default output file name from input file!" + \
> +       stub_msg
> +
>

Re: [PATCH 2/3] Add texinfo-dummy-native recipe w/ scripts to stand in for Texinfo utils.

[Eliaser, MaX]

Sure, I could do that. My understanding of the convention was that there was one directory per upstream project, and all the different recipes in that directory implemented different targets or versions of that upstream project-- is that not correct?

-Max
________________________________________
From: Saul Wold [sgw@linux.intel.com]
Sent: Thursday, May 15, 2014 7:29 AM
To: Eliaser, MaX; openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH 2/3] Add texinfo-dummy-native recipe w/ scripts to stand in for Texinfo utils.

I think these should live in texinfo instead of adding a new
texinfo-dummp-native directory?

Sau!

[PATCH 3/3] native/cross recipes use texinfo-dummy-native; target arch use host's Texinfo.

[Max Eliaser]

To unpack that to more than a single line: -native and -cross recipes are made
to use the dummy Texinfo utilities provided by texinfo-dummy-native if they
invoke those utilities at build time. The target-architecture (cross-compiled)
recipes still use the genuine Texinfo utilites. Right now, they still use
the host system's Texinfo utilities, but could be made to use the
texinfo-native recipe we already ship with some config file changes.

Signed-off-by: Max Eliaser <max.eliaser@intel.com>
---
 meta/classes/texinfo.bbclass | 15 ++++++++++++++-
 meta/conf/bitbake.conf       |  1 +
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/meta/classes/texinfo.bbclass b/meta/classes/texinfo.bbclass
index 60cba0f..92efbcc 100644
--- a/meta/classes/texinfo.bbclass
+++ b/meta/classes/texinfo.bbclass
@@ -1,2 +1,15 @@
 # This class is inherited by recipes whose upstream packages invoke the
-# texinfo utilities at build-time. This class currently does nothing.
+# texinfo utilities at build-time. Native and cross recipes are made to use the
+# dummy scripts provided by texinfo-dummy-native, for improved performance. 
+# Target architecture recipes use the genuine Texinfo utilities. By default, 
+# they use the Texinfo utilities on the host system. If you want to use the
+# Texinfo recipe shipped with yoco, you can remove texinfo-native from 
+# ASSUME_PROVIDED and makeinfo from SANITY_REQUIRED_UTILITIES.
+
+TEXDEP = "texinfo-native"
+TEXDEP_class-native = "texinfo-dummy-native"
+TEXDEP_class-cross = "texinfo-dummy-native"
+DEPENDS_append = " ${TEXDEP}"
+PATH_prepend_class-native = "${STAGING_BINDIR_NATIVE}/texinfo-dummy-native:"
+PATH_prepend_class-cross = "${STAGING_BINDIR_NATIVE}/texinfo-dummy-native:"
+
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 66f435f..889b547 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -180,6 +180,7 @@ ASSUME_PROVIDED = "\
     python-native-runtime \
     tar-native \
     virtual/libintl-native \
+    texinfo-native\
     "
 # gzip-native should be listed above?
 
-- 
1.8.3.2

Re: [PATCH 0/3] Suppress Texinfo utilities invoked at build time for -cross & -native recipes

[Saul Wold]

Max,

we has a failure on the Autobuilder when attempting to build the world 
build.

This has some texinfo related failures, please check them out.

https://autobuilder.yoctoproject.org/main/builders/nightly-world/builds/95/steps/BuildImages/logs/stdio

Thanks

Sau!

On 05/14/2014 02:35 PM, Max Eliaser wrote:
> Hello list,
>
> As per discussion with Richard Purdie and others, I'm submitting a patch to
> suppress some unnecessary invocations of utilities from the Texinfo suite during
> Yocto builds. Specifically, it targets -native and -cross recipes that inherit
> texinfo.bbclass.
>
> The Texinfo utilities are suppressed by supplying doppelgänger executables which
> understand the same command-line options as the genuine Texinfo utilites, and
> which can create blank output files where expected, but which do no actual
> processing, formatting, or markup. To see how this is done, look in the
> texinfo-dummy-native recipe (under meta/recipes-extended,) added in this patch
> series, as well as the modifications to texinfo.bbclass, also in this patch
> series.
>
> Suppressing the Texinfo utilities for -native and -cross recipes does provide a
> noticable performance gain, as documented in bug 2421. [1] You could make the
> case that this patch series still does not completely address bug 2421, as there
> are other utilities not from the Texinfo suite (groff, pod2man, etc) which are
> not suppressed by these changes.
>
> Target-architecture recipes continue to use the host system's Texinfo utilities
> as before, so that dependency is not eliminated by this patch series.
>
> Regards,
> -Max Eliaser
>
> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=2421
> The following changes since commit 7b78c364b7de901f97cd8dbe56d5e02f545bf3d3:
>
>    cleanup-workdir: Fix crosssdk reference after addition of PN (2014-05-05 09:05:42 +0100)
>
> are available in the git repository at:
>
>    git://git.yoctoproject.org/poky-contrib meliaser/texinfo-build
>    http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=meliaser/texinfo-build
>
> Max Eliaser (3):
>    Allow texinfo, texinfo-native recipes to build w/o host system's
>      Texinfo utils
>    Add texinfo-dummy-native recipe w/ scripts to stand in for Texinfo
>      utils.
>    native/cross recipes use texinfo-dummy-native; target arch use host's
>      Texinfo.
>
>   meta/classes/texinfo.bbclass                       |  15 ++-
>   meta/conf/bitbake.conf                             |   1 +
>   .../texinfo-dummy-native/texinfo-dummy-native.bb   |  30 +++++
>   .../texinfo-dummy-native/texinfo-dummy/COPYING     |  19 ++++
>   .../texinfo-dummy-native/texinfo-dummy/template.py | 122 +++++++++++++++++++++
>   .../texinfo/texinfo/disable-native-tools.patch     |  43 --------
>   .../texinfo/texinfo/use_host_makedoc.patch         |  17 ---
>   meta/recipes-extended/texinfo/texinfo_5.2.bb       |   8 +-
>   8 files changed, 188 insertions(+), 67 deletions(-)
>   create mode 100644 meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb
>   create mode 100644 meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/COPYING
>   create mode 100644 meta/recipes-extended/texinfo-dummy-native/texinfo-dummy/template.py
>   delete mode 100644 meta/recipes-extended/texinfo/texinfo/disable-native-tools.patch
>   delete mode 100644 meta/recipes-extended/texinfo/texinfo/use_host_makedoc.patch
>