OE Staging ABI Change Warning

OE Staging ABI Change Warning

[Richard Purdie]

Hi,

OE.dev is now just about ready to switch to using sysroot. The compilers
have been updated to activate sysroot and the fixes I/Poky have for
various pkgconfig issues have been committed. The final change is to
actually change the layout.

I've attached the patch I have locally for doing this. Since it changes
the layout of staging I've gone for changing the ABI and warning users
using the newly added mechanism.

insane.bbclass will warn about potentially problematic .pc files and I
suspect a few people will hit that since there are a lot of broken ones
around. I'll give a summary of the problem and the way to solve it:

The changes activate the sysroot mode of pkgconfig. This means the .pc
files no longer have to be mangled and are quite safe containing
"libdir=/usr/lib" (or whatever $libdir is). pkgconfig itself takes care
of adding the contents of the PKG_CONFIG_SYSROOT_DIR variable
(${STAGING_DIR}/${TARGET_SYS}) to -L and -I options in --libs and
--cflags return values.

The problem comes when apps do broken things with the results from
pkgconfig like adding it to the Libraries line in a .pc file, e.g.:

Libraries: -L/my/path/to/staging

since when pkgconfig returns --libs, you see
-L/my/path/to/staging/my/path/to/staging/ which makes no sense.

The correct fix is usually for apps to add the right package to the
Requires: line, or if the required package doesn't use pkgconfig, use an
expression which uses $libdir.

I'm not likely to be around much over the weekend so I'll hold off
committing this to OE.dev until Monday.

Cheers,

Richard


#
# old_revision [295199ba5efc8ddca9453992b8304f356f385efd]
#
# patch "classes/insane.bbclass"
#  from [00d6311affb65ec365a80a9b833de2945df35660]
#    to [0712df99743c442ed311e6b4d4be0d28f29d525f]
# 
# patch "classes/native.bbclass"
#  from [242e5b275af07bc3c8f5d3d1a6003d852ba72ec6]
#    to [1b40b57b08d22639dfa0d4f7dd05dc5a77f5d82b]
# 
# patch "classes/pkgconfig.bbclass"
#  from [2c935b7d38f7e8274e938105750194e864d60cf7]
#    to [825125168a0e350351266996b570279b35dc175f]
# 
# patch "classes/sdk.bbclass"
#  from [c676bc74ec4f1990541362dca4fe5698aca8eba8]
#    to [3d79f34a1ba3589e0d005c39cdaa6758bf8507f4]
# 
# patch "conf/bitbake.conf"
#  from [ea1e16116ff065e62f58aa7bb32a1757f5aa0695]
#    to [0864432a275e29411be3ab7d16a689c895783c2f]
# 
# patch "conf/sanity.conf"
#  from [19e2e5d9bfa5113b4c25577b445fd588b4a0999b]
#    to [0b8a4c4acfe94430a373d0c7c51dc052f45c98a7]
# 
# patch "packages/gcc/gcc3-build-cross.inc"
#  from [6b61dc663b321ffd6f1992e3587e47c2440c31fe]
#    to [5d07bedf9a0e89893e190e81c7733cbe6d78177b]
# 
# patch "packages/gcc/gcc3-build-sdk.inc"
#  from [e80775fa7cdad10299b39edefff6b5cdb7120bda]
#    to [59b69e7b5420254f92421122abe64c1bafc7ca71]
# 
# patch "packages/gcc/gcc4-build-sdk.inc"
#  from [87829c7148dd1ac08f628d6ee4be53f410490e7b]
#    to [4bccd43b332808cf66e56b0cfcdcc722e35df86c]
# 
# patch "packages/linux-libc-headers/linux-libc-headers.inc"
#  from [87128c42990e29e5ae8b7fb5934036d4c2f46337]
#    to [8d23279cdfe7a7a9d3b20655625b2c6424e25069]
# 
# patch "packages/openssl/openssl.inc"
#  from [e2020c55f48ae6edb2b96f7cb705f900d58ecc79]
#    to [a0a61df4d6fc5943c3fb8c9f885caea1ebd2cb7c]
#
============================================================
--- classes/insane.bbclass	00d6311affb65ec365a80a9b833de2945df35660
+++ classes/insane.bbclass	0712df99743c442ed311e6b4d4be0d28f29d525f
@@ -333,12 +333,15 @@ def package_qa_check_staged(path,d):
     import os, bb
 
     sane = True
-    workdir = os.path.join(bb.data.getVar('TMPDIR', d, True), "work")
+    tmpdir = bb.data.getVar('TMPDIR', d, True)
+    workdir = os.path.join(tmpdir, "work")
 
-    if bb.data.inherits_class("native", d):
+    if bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d):
         installed = "installed=no"
+        pkgconfigcheck = workdir
     else:
         installed = "installed=yes"
+        pkgconfigcheck = tmpdir
 
     # find all .la and .pc files
     # read the content
@@ -356,8 +359,8 @@ def package_qa_check_staged(path,d):
                     sane = package_qa_handle_error(8, error_msg, "staging", path, d)
             elif file[-2:] == "pc":
                 file_content = open(path).read()
-                if workdir in file_content:
-                    error_msg = "%s failed sanity test (workdir) in path %s" % (file,root)
+                if pkgconfigcheck in file_content:
+                    error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
                     sane = package_qa_handle_error(6, error_msg, "staging", path, d)
 
     return sane
============================================================
--- classes/native.bbclass	242e5b275af07bc3c8f5d3d1a6003d852ba72ec6
+++ classes/native.bbclass	1b40b57b08d22639dfa0d4f7dd05dc5a77f5d82b
@@ -47,40 +47,34 @@ export STRIP = "${HOST_PREFIX}strip"
 export RANLIB = "${HOST_PREFIX}ranlib"
 export STRIP = "${HOST_PREFIX}strip"
 
-
 # Path prefixes
-base_prefix = "${STAGING_DIR}/${BUILD_ARCH}-${BUILD_OS}"
-prefix = "${base_prefix}"
-exec_prefix = "${base_prefix}"
+export base_prefix = "${STAGING_DIR_NATIVE}"
+export prefix = "${STAGING_DIR_NATIVE}${layout_prefix}"
+export exec_prefix = "${STAGING_DIR_NATIVE}${layout_exec_prefix}"
 
 # Base paths
-base_bindir = "${base_prefix}/bin"
-base_sbindir = "${base_prefix}/bin"
-base_libdir = "${base_prefix}/lib"
+export base_bindir = "${STAGING_DIR_NATIVE}${layout_base_bindir}"
+export base_sbindir = "${STAGING_DIR_NATIVE}${layout_base_sbindir}"
+export base_libdir = "${STAGING_DIR_NATIVE}${layout_base_libdir}"
 
 # Architecture independent paths
-sysconfdir = "${prefix}/etc"
-sharedstatedir = "${prefix}/com"
-localstatedir = "${prefix}/var"
-infodir = "${datadir}/info"
-mandir = "${datadir}/man"
-docdir = "${datadir}/doc"
-servicedir = "${prefix}/srv"
+export datadir = "${STAGING_DIR_NATIVE}${layout_datadir}"
+export sysconfdir = "${STAGING_DIR_NATIVE}${layout_sysconfdir}"
+export sharedstatedir = "${STAGING_DIR_NATIVE}${layout_sharedstatedir}"
+export localstatedir = "${STAGING_DIR_NATIVE}${layout_localstatedir}"
+export infodir = "${STAGING_DIR_NATIVE}${layout_infodir}"
+export mandir = "${STAGING_DIR_NATIVE}${layout_mandir}"
+export docdir = "${STAGING_DIR_NATIVE}${layout_docdir}"
+export servicedir = "${STAGING_DIR_NATIVE}${layout_servicedir}"
 
 # Architecture dependent paths
-bindir = "${exec_prefix}/bin"
-sbindir = "${exec_prefix}/bin"
-libexecdir = "${exec_prefix}/libexec"
-libdir = "${exec_prefix}/lib"
-includedir = "${exec_prefix}/include"
-oldincludedir = "${exec_prefix}/include"
+export bindir = "${STAGING_DIR_NATIVE}${layout_bindir}"
+export sbindir = "${STAGING_DIR_NATIVE}${layout_sbindir}"
+export libexecdir = "${STAGING_DIR_NATIVE}${layout_libexecdir}"
+export libdir = "${STAGING_DIR_NATIVE}${layout_libdir}"
+export includedir = "${STAGING_DIR_NATIVE}${layout_includedir}"
+export oldincludedir = "${STAGING_DIR_NATIVE}${layout_includedir}"
 
-# Datadir is made arch dependent here, primarily
-# for autoconf macros, and other things that
-# may be manipulated to handle crosscompilation
-# issues.
-datadir = "${exec_prefix}/share"
-
 do_stage () {
 	if [ "${INHIBIT_NATIVE_STAGE_INSTALL}" != "1" ]
 	then
@@ -93,3 +87,4 @@ PKG_CONFIG_PATH .= "${EXTRA_NATIVE_PKGCO
 }
 
 PKG_CONFIG_PATH .= "${EXTRA_NATIVE_PKGCONFIG_PATH}"
+PKG_CONFIG_SYSROOT_DIR = ""
============================================================
--- classes/pkgconfig.bbclass	2c935b7d38f7e8274e938105750194e864d60cf7
+++ classes/pkgconfig.bbclass	825125168a0e350351266996b570279b35dc175f
@@ -1,37 +1,11 @@ DEPENDS_prepend = "pkgconfig-native "
 inherit base
 
 DEPENDS_prepend = "pkgconfig-native "
 
-# The namespaces can clash here hence the two step replace
-def get_pkgconfig_mangle(d):
-	import bb.data
-	s = "-e ''"
-	if not bb.data.inherits_class('native', d):
-		s += " -e 's:=${libdir}:=OELIBDIR:;'"
-		s += " -e 's:=${includedir}:=OEINCDIR:;'"
-		s += " -e 's:=${datadir}:=OEDATADIR:'"
-		s += " -e 's:=${prefix}:=OEPREFIX:'"
-		s += " -e 's:=${exec_prefix}:=OEEXECPREFIX:'"
-		s += " -e 's:OELIBDIR:${STAGING_LIBDIR}:;'"
-		s += " -e 's:OEINCDIR:${STAGING_INCDIR}:;'"
-		s += " -e 's:OEDATADIR:${STAGING_DATADIR}:'"
-		s += " -e 's:OEPREFIX:${STAGING_DIR_HOST}${layout_prefix}:'"
-		s += " -e 's:OEEXECPREFIX:${STAGING_DIR_HOST}${layout_exec_prefix}:'"
-		s += " -e 's:-L${WORKDIR}\S*: :g'"
-		s += " -e 's:-I${WORKDIR}\S*: :g'"
-
-	return s
-
-do_install_append () {
-        for pc in `find ${D} -name '*.pc' -type f | grep -v -- '-uninstalled.pc$'`; do
-                sed -i ${@get_pkgconfig_mangle(d)} -e 's:${D}::g' -e 's:${STAGING_LIBDIR}:${libdir}:g' -e 's:${STAGING_INCDIR}:${includedir}:g' -e 's:${STAGING_DIR_TARGET}:${prefix}:g' ${pc}
-        done
-}
-
 do_stage_append () {
+	install -d ${PKG_CONFIG_DIR}
 	for pc in `find ${S} -name '*.pc' -type f | grep -v -- '-uninstalled.pc$'`; do
 		pcname=`basename $pc`
-		install -d ${PKG_CONFIG_DIR}
-		cat $pc | sed ${@get_pkgconfig_mangle(d)} -e 's:${D}${libdir}\S*:${STAGING_LIBDIR}:g' -e 's:${D}${prefix}/include\S*:${STAGING_INCDIR}:g' > ${PKG_CONFIG_DIR}/$pcname
+		cat $pc > ${PKG_CONFIG_DIR}/$pcname
 	done
 }
============================================================
--- classes/sdk.bbclass	c676bc74ec4f1990541362dca4fe5698aca8eba8
+++ classes/sdk.bbclass	3d79f34a1ba3589e0d005c39cdaa6758bf8507f4
@@ -6,15 +6,15 @@ HOST_ARCH = "${BUILD_ARCH}"
 PACKAGE_ARCH = "${BUILD_ARCH}-${OLD_PACKAGE_ARCH}-sdk"
 
 HOST_ARCH = "${BUILD_ARCH}"
-HOST_VENDOR = "${BUILD_VENDOR}"
+HOST_VENDOR = "${TARGET_VENDOR}"
 HOST_OS = "${BUILD_OS}"
 HOST_PREFIX = "${BUILD_PREFIX}"
 HOST_CC_ARCH = "${BUILD_CC_ARCH}"
 
-CPPFLAGS = "${BUILD_CPPFLAGS}"
-CFLAGS = "${BUILD_CFLAGS}"
-CXXFLAGS = "${BUILD_CFLAGS}"
-LDFLAGS = "${BUILD_LDFLAGS}"
+CPPFLAGS = "${BUILDSDK_CPPFLAGS}"
+CFLAGS = "${BUILDSDK_CFLAGS}"
+CXXFLAGS = "${BUILDSDK_CFLAGS}"
+LDFLAGS = "${BUILDSDK_LDFLAGS}"
 
 # Path prefixes
 prefix = "${SDK_PREFIX}"
@@ -49,3 +49,4 @@ FILES_${PN}-dbg += "${prefix}/.debug \
                     ${prefix}/bin/.debug \
                    "
 
+export PKG_CONFIG_SYSROOT_DIR = "${STAGING_DIR}/${HOST_SYS}"
============================================================
--- conf/bitbake.conf	ea1e16116ff065e62f58aa7bb32a1757f5aa0695
+++ conf/bitbake.conf	0864432a275e29411be3ab7d16a689c895783c2f
@@ -1,24 +1,21 @@
 ##################################################################
 # Standard target filesystem layout.
 ##################################################################
 
-# Note these currently match the existing staging layout but this 
-# is planned to change, see the oe-dev mailing list
-
 # Path prefixes
-layout_prefix = ""
-layout_exec_prefix = ""
+layout_prefix = "/usr"
+layout_exec_prefix = "/usr"
 layout_base_prefix = ""
 
 # Base paths
 layout_base_bindir = "${layout_base_prefix}/bin"
-layout_base_sbindir = "${layout_base_prefix}/bin"
+layout_base_sbindir = "${layout_base_prefix}/sbin"
 layout_base_libdir = "${layout_base_prefix}/lib"
 
 # Architecture independent paths
-layout_sysconfdir = "/etc"
-layout_localstatedir = "/var"
-layout_servicedir = "/srv"
+layout_sysconfdir = "${layout_base_prefix}/etc"
+layout_localstatedir = "${layout_base_prefix}/var"
+layout_servicedir = "${layout_base_prefix}/srv"
 layout_sharedstatedir = "${layout_prefix}/com"
 layout_datadir = "${layout_prefix}/share"
 layout_infodir = "${layout_datadir}/info"
@@ -27,7 +24,7 @@ layout_bindir = "${layout_exec_prefix}/b
 
 # Architecture dependent paths
 layout_bindir = "${layout_exec_prefix}/bin"
-layout_sbindir = "${layout_exec_prefix}/bin"
+layout_sbindir = "${layout_exec_prefix}/sbin"
 layout_libdir = "${layout_exec_prefix}/lib"
 layout_includedir = "${layout_exec_prefix}/include"
 layout_libexecdir = "${layout_exec_prefix}/libexec"
@@ -38,31 +35,31 @@ export base_prefix = ""
 
 # Path prefixes
 export base_prefix = ""
-export prefix = "/usr"
-export exec_prefix = "${prefix}"
+export prefix = "${layout_prefix}"
+export exec_prefix = "${layout_exec_prefix}"
 
 # Base paths
-export base_bindir = "${base_prefix}/bin"
-export base_sbindir = "${base_prefix}/sbin"
-export base_libdir = "${base_prefix}/lib"
+export base_bindir = "${layout_base_bindir}"
+export base_sbindir = "${layout_base_sbindir}"
+export base_libdir = "${layout_base_libdir}"
 
 # Architecture independent paths
-export datadir = "${prefix}/share"
-export sysconfdir = "/etc"
-export sharedstatedir = "${prefix}/com"
-export localstatedir = "/var"
-export infodir = "${datadir}/info"
-export mandir = "${datadir}/man"
-export docdir = "${datadir}/doc"
-export servicedir = "/srv"
+export datadir = "${layout_datadir}"
+export sysconfdir = "${layout_sysconfdir}"
+export sharedstatedir = "${layout_sharedstatedir}"
+export localstatedir = "${layout_localstatedir}"
+export infodir = "${layout_infodir}"
+export mandir = "${layout_mandir}"
+export docdir = "${layout_docdir}"
+export servicedir = "${layout_servicedir}"
 
 # Architecture dependent paths
-export bindir = "${exec_prefix}/bin"
-export sbindir = "${exec_prefix}/sbin"
-export libexecdir = "${exec_prefix}/libexec"
-export libdir = "${exec_prefix}/lib"
-export includedir = "${exec_prefix}/include"
-export oldincludedir = "${exec_prefix}/include"
+export bindir = "${layout_bindir}"
+export sbindir = "${layout_sbindir}"
+export libexecdir = "${layout_libexecdir}"
+export libdir = "${layout_libdir}"
+export includedir = "${layout_includedir}"
+export oldincludedir = "${layout_includedir}"
 
 ##################################################################
 # Architecture-dependent build variables.
@@ -506,6 +503,7 @@ export PKG_CONFIG_PATH = "${PKG_CONFIG_D
 
 export PKG_CONFIG_DIR = "${STAGING_LIBDIR}/pkgconfig"
 export PKG_CONFIG_PATH = "${PKG_CONFIG_DIR}"
+export PKG_CONFIG_SYSROOT_DIR = "${STAGING_DIR}/${TARGET_SYS}"
 export PKG_CONFIG_DISABLE_UNINSTALLED = "yes"
 
 export QMAKE_MKSPEC_PATH = "${STAGING_DATADIR_NATIVE}/qmake"
============================================================
--- conf/sanity.conf	19e2e5d9bfa5113b4c25577b445fd588b4a0999b
+++ conf/sanity.conf	0b8a4c4acfe94430a373d0c7c51dc052f45c98a7
@@ -11,7 +11,7 @@ BB_MIN_VERSION = "1.8.10"
 # that breaks the format and have been previously discussed on the mailing list 
 # with general agreement from the core team.
 #
-SANITY_ABI = "0"
+SANITY_ABI = "1"
 SANITY_ABIFILE = "${TMPDIR}/abi_version"
 
 INHERIT += "sanity"
============================================================
--- packages/gcc/gcc3-build-cross.inc	6b61dc663b321ffd6f1992e3587e47c2440c31fe
+++ packages/gcc/gcc3-build-cross.inc	5d07bedf9a0e89893e190e81c7733cbe6d78177b
@@ -23,11 +23,6 @@ do_stage_append () {
 		rm -rf ${CROSS_DIR}/$d
 	done
 
-    # Hack to be removed when staging layout matches the target layout
-    if [ ! -e ${STAGING_DIR_TARGET}/usr ]; then
-        ln -s ${STAGING_DIR_TARGET} ${STAGING_DIR_TARGET}/usr
-    fi
-
 	# Fix a few include links so cross builds are happier
 	if [ ! -e ${STAGING_INCDIR}/c++ ]; then
 		mkdir -p ${STAGING_INCDIR}
============================================================
--- packages/gcc/gcc3-build-sdk.inc	e80775fa7cdad10299b39edefff6b5cdb7120bda
+++ packages/gcc/gcc3-build-sdk.inc	59b69e7b5420254f92421122abe64c1bafc7ca71
@@ -5,6 +5,8 @@ STAGING_TARGET_LIBDIR = "${STAGING_DIR_T
 STAGING_TARGET_INCDIR = "${STAGING_DIR_TARGET}${layout_includedir}"
 STAGING_TARGET_LIBDIR = "${STAGING_DIR_TARGET}${layout_libdir}"
 
+EXTRA_OECONF += "--with-sysroot=${prefix}/${TARGET_SYS} --with-build-sysroot=${STAGING_DIR_TARGET}"
+
 do_configure () {
        (cd ${S} && gnu-configize) || die "failure running gnu-configize"
        (cd ${S}/libstdc++-v3 && autoreconf)
============================================================
--- packages/gcc/gcc4-build-sdk.inc	87829c7148dd1ac08f628d6ee4be53f410490e7b
+++ packages/gcc/gcc4-build-sdk.inc	4bccd43b332808cf66e56b0cfcdcc722e35df86c
@@ -1,13 +1,9 @@ do_configure () {
 USE_NLS = '${@base_conditional( "TARGET_OS", "linux-uclibc", "no", "", d )}'
 USE_NLS = '${@base_conditional( "TARGET_OS", "linux-uclibcgnueabi", "no", "", d )}'
 
 EXTRA_OECONF += "--with-sysroot=${prefix}/${TARGET_SYS} --with-build-sysroot=${STAGING_DIR_TARGET}"
 
 do_configure () {
-       # Hack to be removed when staging layout matches the target layout
-       if [ ! -e ${STAGING_DIR_TARGET}/usr ]; then
-           ln -s ${STAGING_DIR_TARGET} ${STAGING_DIR_TARGET}/usr
-       fi
        (cd ${S} && gnu-configize) || die "failure running gnu-configize"
        (cd ${S}/libstdc++-v3 && autoreconf)
        oe_runconf
============================================================
--- packages/linux-libc-headers/linux-libc-headers.inc	87128c42990e29e5ae8b7fb5934036d4c2f46337
+++ packages/linux-libc-headers/linux-libc-headers.inc	8d23279cdfe7a7a9d3b20655625b2c6424e25069
@@ -1,5 +1,3 @@ LICENSE = "GPL"
 DESCRIPTION = "Sanitized set of 2.6 kernel headers for the C library's use."
 SECTION = "devel"
 LICENSE = "GPL"
-
-DEPENDS = "cross-linkage"
============================================================
--- packages/openssl/openssl.inc	e2020c55f48ae6edb2b96f7cb705f900d58ecc79
+++ packages/openssl/openssl.inc	a0a61df4d6fc5943c3fb8c9f885caea1ebd2cb7c
@@ -14,7 +14,7 @@ export DIRS = "crypto ssl apps"
 export CFLAG_mtx-2 := "${@'${CFLAG}'.replace('-O2', '')}"
 
 export DIRS = "crypto ssl apps"
-export EX_LIBS = "-lgcc -ldl ${TARGET_LDFLAGS}"
+export EX_LIBS = "-lgcc -ldl"
 export AS = "${CC} -c"
 
 PACKAGES =+ "libcrypto libssl"

Re: OE Staging ABI Change Warning

[Leon Woestenberg]

Richard,

excellent work and plan of attack.

The patch looks quite sane to me (although I'm not familiar with all
the pesky little details), and found one thing that I couldn't grasp:

+++ classes/sdk.bbclass 3d79f34a1ba3589e0d005c39cdaa6758bf8507f4
@@ -6,15 +6,15 @@ HOST_ARCH = "${BUILD_ARCH}"
 PACKAGE_ARCH = "${BUILD_ARCH}-${OLD_PACKAGE_ARCH}-sdk"

 HOST_ARCH = "${BUILD_ARCH}"
-HOST_VENDOR = "${BUILD_VENDOR}"
+HOST_VENDOR = "${TARGET_VENDOR}"

Could you explain why this must be TARGET_VENDOR instead of BUILD_VENDOR?

Thanks, regards,

-- 
Leon 'likewise'

Re: OE Staging ABI Change Warning

[Richard Purdie]

On Fri, 2008-02-29 at 18:24 +0100, Leon Woestenberg wrote:
> excellent work and plan of attack.

Thanks :)

> The patch looks quite sane to me (although I'm not familiar with all
> the pesky little details), and found one thing that I couldn't grasp:
> 
> +++ classes/sdk.bbclass 3d79f34a1ba3589e0d005c39cdaa6758bf8507f4
> @@ -6,15 +6,15 @@ HOST_ARCH = "${BUILD_ARCH}"
>  PACKAGE_ARCH = "${BUILD_ARCH}-${OLD_PACKAGE_ARCH}-sdk"
> 
>  HOST_ARCH = "${BUILD_ARCH}"
> -HOST_VENDOR = "${BUILD_VENDOR}"
> +HOST_VENDOR = "${TARGET_VENDOR}"
> 
> Could you explain why this must be TARGET_VENDOR instead of BUILD_VENDOR?

Its a little bit hacky but there is a good reason. Basically you don't
want -native and -sdk packages in the same directory. If you don't
change that here, they will trample on each other in staging.

Poky's currently using that as a workaround and Poky makes the most
extensive use of -sdk packages the metadata has seen so its better than
the current position in OE.dev. I don't claim its the best solution
though ;-).

Cheers,

Richard

Re: OE Staging ABI Change Warning

[Leon Woestenberg]

Hello,

On Fri, Feb 29, 2008 at 7:04 PM, Richard Purdie <rpurdie@rpsys.net> wrote:
> On Fri, 2008-02-29 at 18:24 +0100, Leon Woestenberg wrote:
>  > excellent work and plan of attack.
>
>  > Could you explain why this must be TARGET_VENDOR instead of BUILD_VENDOR?
>
>  Its a little bit hacky but there is a good reason. Basically you don't
>  want -native and -sdk packages in the same directory. If you don't
>
Could we just add a comment explaining this approach above the assignment?
Everything that is not trivial by code inspection surely deserves a comment.

I recon I could try the patch already.

Regards,
-- 
Leon

Re: OE Staging ABI Change Warning

[Richard Purdie]

On Fri, 2008-02-29 at 19:29 +0100, Leon Woestenberg wrote:
> On Fri, Feb 29, 2008 at 7:04 PM, Richard Purdie <rpurdie@rpsys.net> wrote:
> > On Fri, 2008-02-29 at 18:24 +0100, Leon Woestenberg wrote:
> >  > excellent work and plan of attack.
> >
> >  > Could you explain why this must be TARGET_VENDOR instead of BUILD_VENDOR?
> >
> >  Its a little bit hacky but there is a good reason. Basically you don't
> >  want -native and -sdk packages in the same directory. If you don't
> >
> Could we just add a comment explaining this approach above the assignment?
> Everything that is not trivial by code inspection surely deserves a comment.

Agreed, I will add something before I commit it.

> I recon I could try the patch already.

I'm happy for anyone to do so, its been tested locally so does basically
work although as I'm sure is clear, this is a fundamental change and
some things will break :/. 

FWIW, its what Poky has been using for a while so the 'GMAE stack' is
well tested, its the areas outside that which I worry about.

Cheers,

Richard

Re: OE Staging ABI Change Warning

[Holger Freyther]

On Friday 29 February 2008 17:46:04 Richard Purdie wrote:
> Hi,
>
> OE.dev is now just about ready to switch to using sysroot. The compilers
> have been updated to activate sysroot and the fixes I/Poky have for
> various pkgconfig issues have been committed. The final change is to
> actually change the layout.

Hi Richard,

one small question, how does this impact:
	-The sharp rom and their crappy and ancient 2.95.x toolchain?
	-Any other prebuilt toolchain.

and honestly I'm a bit scared of silently breaking thousands of packages/BB 
files.


regards
	z.

Re: OE Staging ABI Change Warning

[Richard Purdie]

Hi Holger,

On Mon, 2008-03-03 at 02:07 +0100, Holger Freyther wrote:
> On Friday 29 February 2008 17:46:04 Richard Purdie wrote:
> > OE.dev is now just about ready to switch to using sysroot. The compilers
> > have been updated to activate sysroot and the fixes I/Poky have for
> > various pkgconfig issues have been committed. The final change is to
> > actually change the layout.
>
> one small question, how does this impact:
> 	-The sharp rom and their crappy and ancient 2.95.x toolchain?
> 	-Any other prebuilt toolchain.

It means they'll need to play a game with symlinks just as we currently
have in cross-linkage and the "hack" I added into gcc-cross temporarily
to add a usr symlink in staging.

With the addition of those two symlinks, OE should continue to work for
all old toolchains and prebuilt ones without sysroot support.

> and honestly I'm a bit scared of silently breaking thousands of packages/BB 
> files.

Having looked at some old parts of the metadata when fixing up staging
references, we already have a ton of broken .bb files :(. I was really
disappointed with the quality in some areas and I honestly don't know
what to do about this.


If it helps, I don't think this stage of the conversion is that risky.
The main problems will arise from the pkgconfig change and insane will
instantly pick those problems up since they are clearly identifiable
(and not silent). Its the pkgconfig change that is breaking staging ABI,
the actual layout change could have been made with symlinks (I kind of
already did that) and I doubt many would have noticed.

The really nasty/risky conversion would be if, as and when we change the
TARGET_CFLAGS and TARGET_LDFLAGS to not include staging paths and rely
solely on sysroot. This worked in Poky but did cause some headaches and
I'm not going to propose it for OE.dev at this time. The nice thing
about it is we can do it without breaking ABI and toolchains/distros
that need the extra flags can just add them back in so its a much less
problematic change in that sense.

Cheers,

Richard

Re: OE Staging ABI Change Warning

[Holger Freyther]

On Monday 03 March 2008 10:37:59 Richard Purdie wrote:
> Hi Holger,
>
> On Mon, 2008-03-03 at 02:07 +0100, Holger Freyther wrote:
> > On Friday 29 February 2008 17:46:04 Richard Purdie wrote:
> > > OE.dev is now just about ready to switch to using sysroot. The
> > > compilers have been updated to activate sysroot and the fixes I/Poky
> > > have for various pkgconfig issues have been committed. The final change
> > > is to actually change the layout.
> >
> > one small question, how does this impact:
> > 	-The sharp rom and their crappy and ancient 2.95.x toolchain?
> > 	-Any other prebuilt toolchain.
>
> It means they'll need to play a game with symlinks just as we currently
> have in cross-linkage and the "hack" I added into gcc-cross temporarily
> to add a usr symlink in staging.

This needs documentation (updates). I don't see any of us volunteering though.


> Having looked at some old parts of the metadata when fixing up staging
> references, we already have a ton of broken .bb files :(. I was really
> disappointed with the quality in some areas and I honestly don't know
> what to do about this.

There is proposal to start with an empty repository and move things over, 
testing and compiling world, clean up weekends.... more review on checking 
things in.

Back to the point, feel free to check things in.


z.

Re: OE Staging ABI Change Warning

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Holger Freyther schreef:
| On Monday 03 March 2008 10:37:59 Richard Purdie wrote:

|> Having looked at some old parts of the metadata when fixing up staging
|> references, we already have a ton of broken .bb files :(. I was really
|> disappointed with the quality in some areas and I honestly don't know
|> what to do about this.
|
| There is proposal to start with an empty repository and move things over,
| testing and compiling world, clean up weekends.... more review on
checking
| things in.

The previous consensus to that proposal was "we should be able to do
cleanups in-place". I still think we should start such a branch where we
can do the clean up while retaining the history. Those cleanups would
consist of:

a) basic janitorial stuff:

* running the recipe through oe-stylize.py and fix accordingly
* document patches (origin, upstream status, etc)
* restucrure FILESDIR to avoid duplicate files (e.g. files/foo.patch
bar-1.0/foo.patch, etc)

b) updating to OE standards

* Replace custom methods with standards ones where appropriate (e.g.
do_stage() -> autotools_stage_all
* document remaining custom methods (e.g. do_configure_append sed magic)
* sanitize PACKAGES and FILES
* sanitize overrides (e.g. fix all the CONFFILES_nylon stuff)

c) unit testing

* compile for different archs
* run through insane

Regardless if we create a branch for that or not, we should declare a
freeze for e.g. 3 months where only bugfixes and cleanups are allowed to
be checked in.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFHy/NCMkyGM64RGpERAkjlAJ9GgDhax6XcSE0kQja3za3fxK8e6wCZAdba
6G2Uq14grE9WgmwjTfjqGMs=
=vD/7
-----END PGP SIGNATURE-----

Re: OE Staging ABI Change Warning

[Richard Purdie]

On Mon, 2008-03-03 at 13:46 +0100, Koen Kooi wrote:
> Holger Freyther schreef:
> The previous consensus to that proposal was "we should be able to do
> cleanups in-place". I still think we should start such a branch where we
> can do the clean up while retaining the history. Those cleanups would
> consist of:
> 
> a) basic janitorial stuff:
> 
> * running the recipe through oe-stylize.py and fix accordingly
> * document patches (origin, upstream status, etc)
> * restucrure FILESDIR to avoid duplicate files (e.g. files/foo.patch
> bar-1.0/foo.patch, etc)
> 
> b) updating to OE standards
> 
> * Replace custom methods with standards ones where appropriate (e.g.
> do_stage() -> autotools_stage_all
> * document remaining custom methods (e.g. do_configure_append sed magic)
> * sanitize PACKAGES and FILES
> * sanitize overrides (e.g. fix all the CONFFILES_nylon stuff)
> 
> c) unit testing
> 
> * compile for different archs
> * run through insane
> 
> Regardless if we create a branch for that or not, we should declare a
> freeze for e.g. 3 months where only bugfixes and cleanups are allowed to
> be checked in.

I like the idea of a cleanup process, I don't like the ideas of
branching or freezing since I think we don't have the manpower for it.

With this in mind I'd propose we just mark clean/active recipes somehow,
probably by setting a variable in them. We can then work out which
recipes have been looked at and which haven't which is the current
problem at the moment. Including a date in this variable might be good. 

One thing I did with Poky's autobuilder to improve QA was activating
world builds. We now have a list of known broken recipes in the
distro.conf file which excludes the problems from the world builds (with
reasons why). When people check anything further into Poky, we get to
know if it breaks anything "known to build" very quickly. Obviously I
aim to have a list of zero excludes eventually.

So, I guess what I'm saying is I'd like to see regular world builds of
OE with say Angstrom as a distro since it has the widest coverage and a
list of known failures created. Any new failures can then be identified
and both these and existing failures can be addressed over time.

I know angstrom is doing some kind of autobuilds but I don't know to
what extent or what policies govern what is built. I also know the nslu2
people maintain lists of packages to build and not to build. If we could
somehow pool this information that would be ideal.

I'm also giving serious consideration to expanding the suite of qemu
machines like qemuarm/qemux86 so even people without access to hardware
can run tests on the hardware QEMU supports. Poky uses the qemu machines
for the world builds on the basis that they make a good test target in
that if they build, most other things probably will too. The main thing
preventing me from doing this is time at the moment.

Cheers,

Richard

Re: OE Staging ABI Change Warning

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Richard Purdie schreef:

| I know angstrom is doing some kind of autobuilds but I don't know to
| what extent or what policies govern what is built.

The autobuilders run contrib/angstrom/build-*.sh on a regular interval
and sepukku.bbclass takes care of the reporting. I'd like to work on
improving that, but can't since my bugzilla privs disappeared under
suspicious circumstances....

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFHzA5+MkyGM64RGpERArv6AJ4tFkk5b8AcFjpOPbQTXlKkZhr/uACgi2At
8CELjgaM4amqZvAkBG/soAM=
=WDhm
-----END PGP SIGNATURE-----

Re: OE Staging ABI Change Warning

[Rod Whitby]

Richard Purdie wrote:
> I know angstrom is doing some kind of autobuilds but I don't know to
> what extent or what policies govern what is built. I also know the nslu2
> people maintain lists of packages to build and not to build. If we could
> somehow pool this information that would be ideal.

The NSLU2-Linux people would be happy to participate in a pooling of 
marking broken and working packages.

Our current list is at packages/meta/slugos-packages.bb and that list is 
continually incrementally autobuilt on the nslu2-linux autobuilder.

-- Rod

Re: OE Staging ABI Change Warning

[Leon Woestenberg]

Hello all,


I assume, the broken packages usually are due to developers now
knowing the details of what is fully correct, and subsequent
copy-pasting.

So, please let's make this a learning event:

* If you notice something broken, not only fix it, but explain *why*
its broken (here, on the ml?)
* Also, we should check if the documentation covered the issue in the
first place.

If we covered a few issues which are understood by developers, I think
most developers are glad to review their own packages.

Regards,
-- 
Leon

Re: OE Staging ABI Change Warning

[Leon Woestenberg]

Correction,

On Mon, Mar 3, 2008 at 5:14 PM, Leon Woestenberg
<leon.woestenberg@gmail.com> wrote:
> Hello all,
>
>
>  I assume, the broken packages usually are due to developers now
>  knowing the details of what is fully correct, and subsequent
>  copy-pasting.
>
I assume, the broken packages usually are due to developers *not*
knowing the details of what is fully correct, and subsequent
copy-pasting.

Regards,
-- 
Leon

Re: OE Staging ABI Change Warning

[Mike (mwester)]

Richard Purdie wrote:
> Hi,
>
> OE.dev is now just about ready to switch to using sysroot. The compilers
> have been updated to activate sysroot and the fixes I/Poky have for
> various pkgconfig issues have been committed....

Unslung now fails when building gcc-cross.  I presume this is a result 
of this activity (it build before the Feb 28 commits), but I do not 
understand the issues about sysroot and all well enough to even begin to 
be able to troubleshoot.  Is the patch attached to this original email 
supposed to fix this?  Is there something broken in the recipe, as 
implied by this thread, that I need to fix?  If so, what would that be?

[skipping over the normal compilation output in the logs...]
gcc -DIN_GCC -DHAVE_CONFIG_H 
-isystem/stuff/slug/unslung/tmp/staging/i686-linux/include -O2 -DIN_GCC 
-DCROSS_\
COMPILE -W -Wall -Wwrite-strings -Wstrict-prototypes 
-Wmissing-prototypes -pedantic -Wno-long-long -Wno-error\
 -DHAVE_CONFIG_H -DGENERATOR_FILE 
-isystem/stuff/slug/unslung/tmp/staging/i686-linux/include -W -Wall -Wwrite\
-strings -Wstrict-prototypes -Wmissing-prototypes -pedantic 
-Wno-long-long  -Wno-error -I. -I.. -I/stuff/slug\
/unslung/tmp/work/gcc-cross-3.4.4-r8/gcc-3.4.4/gcc/fixinc 
-I/stuff/slug/unslung/tmp/work/gcc-cross-3.4.4-r8/g\
cc-3.4.4/gcc/fixinc/.. 
-I/stuff/slug/unslung/tmp/work/gcc-cross-3.4.4-r8/gcc-3.4.4/gcc/fixinc/../config 
-I/st\
uff/slug/unslung/tmp/work/gcc-cross-3.4.4-r8/gcc-3.4.4/gcc/fixinc/../../include  
-o fixincl fixincl.o fixtest\
s.o fixfixes.o server.o procopen.o fixlib.o 
/stuff/slug/unslung/tmp/work/gcc-cross-3.4.4-r8/gcc-3.4.4/build.i\
686-linux.armeb-linux/gcc/../libiberty/libiberty.a
echo timestamp > full-stamp
./fixincl -v < /dev/null
'fixincl version 1.1'
chmod 755 ../fixinc.sh
make[4]: Leaving directory 
`/stuff/slug/unslung/tmp/work/gcc-cross-3.4.4-r8/gcc-3.4.4/build.i686-linux.armeb-\
linux/gcc/fixinc'
The directory that should contain system headers does not exist:
  /stuff/slug/unslung/tmp/staging/armeb-linux/usr/include
make[3]: *** [stmp-fixinc] Error 1
make[3]: Leaving directory 
`/stuff/slug/unslung/tmp/work/gcc-cross-3.4.4-r8/gcc-3.4.4/build.i686-linux.armeb-\
linux/gcc'
make[2]: *** [all-gcc] Error 2
make[2]: Leaving directory 
`/stuff/slug/unslung/tmp/work/gcc-cross-3.4.4-r8/gcc-3.4.4/build.i686-linux.armeb-\
linux'
FATAL: oe_runmake failed



In particular, note the error message which claims that the 
...gcc-3.4.4/usr/include file does not exist.  No, it does not -- it's 
at ...gcc-3.4.4/include instead.

Thanks,
Mike (mwester)

Re: OE Staging ABI Change Warning

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Richard Purdie schreef:
| Hi,
|
| OE.dev is now just about ready to switch to using sysroot. The compilers
| have been updated to activate sysroot and the fixes I/Poky have for
| various pkgconfig issues have been committed. The final change is to
| actually change the layout.

I decided now is a good time to commit testlab.bbclass so people can
track down *image* differences quite easily before and after these
changes. Just add INHERIT += "testlab" to your local.conf and read the
comments in classes/testlab.bbclass to see what it does.

This was done in an effort to give everyone a CELF testlab at home, with
target regression testing a planned feature[1].

regards,

Koen

[1] Currently depends on how fast USPS can deliver to .nl and wether the
products from http://20sim.com are stable enough to finish my Control
Engineering assigments in a timely fashion ;)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFHzJMVMkyGM64RGpERApVYAJ9bzjwoAS2ltORm3CTgLvMePp8fZgCeIl39
JPfYgVZjKMXeVnGCpIpa11k=
=Q2h6
-----END PGP SIGNATURE-----

Re: OE Staging ABI Change Warning

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Richard Purdie schreef:
| Hi,
|
| OE.dev is now just about ready to switch to using sysroot. The compilers
| have been updated to activate sysroot and the fixes I/Poky have for
| various pkgconfig issues have been committed. The final change is to
| actually change the layout.
|
| I've attached the patch I have locally for doing this. Since it changes
| the layout of staging I've gone for changing the ABI and warning users
| using the newly added mechanism.
|
| insane.bbclass will warn about potentially problematic .pc files and I
| suspect a few people will hit that since there are a lot of broken ones
| around. I'll give a summary of the problem and the way to solve it:

When are we going to make insane halt on .la files referring to
${WORKDIR}? Right now those are the biggest problem for packaged-staging
and rm_work.

regards,

Koen
- --
koen@dominion.kabel.utwente.nl will go go away in december 2007, please
use k.kooi@student.utwente.nl instead.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFHzX6DMkyGM64RGpERAuy8AJ4xZS1LJgLstnz6NjEg3Tw3QPkHXgCcDBUw
Ffldpcorfw2FpBT52t9YfWA=
=btjD
-----END PGP SIGNATURE-----

Re: OE Staging ABI Change Warning

[Richard Purdie]

On Tue, 2008-03-04 at 17:53 +0100, Koen Kooi wrote:
> Richard Purdie schreef:
> | Hi,
> |
> | OE.dev is now just about ready to switch to using sysroot. The compilers
> | have been updated to activate sysroot and the fixes I/Poky have for
> | various pkgconfig issues have been committed. The final change is to
> | actually change the layout.
> |
> | I've attached the patch I have locally for doing this. Since it changes
> | the layout of staging I've gone for changing the ABI and warning users
> | using the newly added mechanism.
> |
> | insane.bbclass will warn about potentially problematic .pc files and I
> | suspect a few people will hit that since there are a lot of broken ones
> | around. I'll give a summary of the problem and the way to solve it:
> 
> When are we going to make insane halt on .la files referring to
> ${WORKDIR}? Right now those are the biggest problem for packaged-staging
> and rm_work.

I don't remember seeing too many warnings about these? Are there many
problematic packages?

Cheers,

Richard

Re: OE Staging ABI Change Warning

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Richard Purdie schreef:
| On Tue, 2008-03-04 at 17:53 +0100, Koen Kooi wrote:
|> Richard Purdie schreef:
|> | Hi,
|> |
|> | OE.dev is now just about ready to switch to using sysroot. The
compilers
|> | have been updated to activate sysroot and the fixes I/Poky have for
|> | various pkgconfig issues have been committed. The final change is to
|> | actually change the layout.
|> |
|> | I've attached the patch I have locally for doing this. Since it changes
|> | the layout of staging I've gone for changing the ABI and warning users
|> | using the newly added mechanism.
|> |
|> | insane.bbclass will warn about potentially problematic .pc files and I
|> | suspect a few people will hit that since there are a lot of broken ones
|> | around. I'll give a summary of the problem and the way to solve it:
|>
|> When are we going to make insane halt on .la files referring to
|> ${WORKDIR}? Right now those are the biggest problem for packaged-staging
|> and rm_work.
|
| I don't remember seeing too many warnings about these? Are there many
| problematic packages?

Almost every native package suffers from it, which is why the efl stuff
fails, since that is needing the native evas, edje, etc tools.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFHz+p4MkyGM64RGpERAqA+AKCIBCY03/myptRoMYuB5S1/KOoNrwCginMe
gD2gozgUxlmFAPcrPJEY/z0=
=uG24
-----END PGP SIGNATURE-----