* [PATCH v5 00/10] support profile-optimized build for Python
@ 2017-10-27 10:07 Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 01/10] python3-native: support profile optimized build Markus Lehtonen
` (12 more replies)
0 siblings, 13 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-10-27 10:07 UTC (permalink / raw)
To: openembedded-core
Fifth iteration of my patchset for enabling profile-guided-optimization for
Python3.
Changes since v4:
- Rebased on latest oe-core
- Usage of EXCLUDE_FROM_SHLIBS in python3-profile-opt is dropped, in favor of
renaming of libpython3 in python3-profile-opt (re-introduced as this was
dropped in v3). Need this so that the shlib dependencies are correctly tracked,
and at the same time, soname clashes between libpython3 and
libpython3-profile-opt are prevented.
- Patch bitbake.conf to include ip and scp in HOSTTOOLS. These are required to
run the profile task on device.
The following changes since commit 3b413a80578caacd9a7f405f3c51a3921d78a60d:
README.qemu: qemuppc64 is not supported (2017-10-16 23:54:27 +0100)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib marquiz/fixes-9338
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=marquiz/fixes-9338
Markus Lehtonen (10):
python3-native: support profile optimized build
python3: fix depends of python3-tests
python3: add python3-profile-opt recipe
python3-profile-opt: rename libpython3
bitbake.conf: add ip and scp to HOSTTOOLS unconditionally
devtools/images: add python-pgo-image
python3: support profile optimized build
python3: fix profile-optimized build of modules
python3: add python3-tools subpackage
python3: remove two setup.py cross-compile hacks
meta/conf/bitbake.conf | 6 +-
meta/recipes-devtools/images/python-pgo-image.bb | 92 ++++++++++++++++++
.../python/python-3.5-manifest.inc | 10 +-
.../python/python3-native_3.5.3.bb | 9 ++
meta/recipes-devtools/python/python3-profile-opt | 1 +
.../python/python3-profile-opt_3.5.3.bb | 10 ++
.../python3/0001-cross-compile-support.patch | 9 --
...efile-add-install_generate_profile-target.patch | 25 +++++
...-CFLAGS-for-extensions-when-cross-compili.patch | 56 +++++++++++
.../python3-setup.py-no-host-headers-libs.patch | 33 -------
...name-libpython3-to-libpython-profile-opt3.patch | 108 +++++++++++++++++++++
.../setup.py-find-libraries-in-staging-dirs.patch | 30 ------
meta/recipes-devtools/python/python3_3.5.3.bb | 58 ++++++++---
13 files changed, 352 insertions(+), 95 deletions(-)
create mode 100644 meta/recipes-devtools/images/python-pgo-image.bb
create mode 120000 meta/recipes-devtools/python/python3-profile-opt
create mode 100644 meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
create mode 100644 meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch
create mode 100644 meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch
delete mode 100644 meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch
create mode 100644 meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch
delete mode 100644 meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
--
2.13.6
^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v5 01/10] python3-native: support profile optimized build
2017-10-27 10:07 [PATCH v5 00/10] support profile-optimized build for Python Markus Lehtonen
@ 2017-10-27 10:07 ` Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 02/10] python3: fix depends of python3-tests Markus Lehtonen
` (11 subsequent siblings)
12 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-10-27 10:07 UTC (permalink / raw)
To: openembedded-core
Make it possible to build python3-native with profile directed optimization
enabled. The feature is enabled by specifying
PYTHON3_NATIVE_PROFILE_OPT = "1" in local.conf.
The profile task to be run may be specified with PYTHON3_NATIVE_PROFILE_TASK
variable in local.conf, e.g.
PYTHON3_NATIVE_PROFILE_TASK = "${S}/Tools/pybench/pybench.py -n 2 --with-gc --with-syscheck"
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python3-native_3.5.3.bb | 9 +++++++++
.../python/python3/0001-cross-compile-support.patch | 9 ---------
2 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/meta/recipes-devtools/python/python3-native_3.5.3.bb b/meta/recipes-devtools/python/python3-native_3.5.3.bb
index 8cd9c88a82..936ea05b29 100644
--- a/meta/recipes-devtools/python/python3-native_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3-native_3.5.3.bb
@@ -58,11 +58,20 @@ EXTRA_OEMAKE = '\
# No ctypes option for python 3
PYTHONLSBOPTS = ""
+PYTHON3_NATIVE_MAKE_TARGET ?= "${@'profile-opt' if d.getVar('PYTHON3_NATIVE_PROFILE_OPT', True) == '1' else ''}"
+
do_configure_append() {
autoreconf --verbose --install --force --exclude=autopoint ../Python-${PV}/Modules/_ctypes/libffi
sed -i -e 's,#define HAVE_GETRANDOM 1,/\* #undef HAVE_GETRANDOM \*/,' ${B}/pyconfig.h
}
+do_compile() {
+ if [ -n "${PYTHON3_NATIVE_PROFILE_TASK}" ]; then
+ sed -i -e 's,^PROFILE_TASK=.*,PROFILE_TASK=${PYTHON3_NATIVE_PROFILE_TASK},g' Makefile
+ fi
+ oe_runmake ${PYTHON3_NATIVE_MAKE_TARGET}
+}
+
do_install() {
install -d ${D}${libdir}/pkgconfig
oe_runmake 'DESTDIR=${D}' install
diff --git a/meta/recipes-devtools/python/python3/0001-cross-compile-support.patch b/meta/recipes-devtools/python/python3/0001-cross-compile-support.patch
index 118d75ddc5..bd599a8b7e 100644
--- a/meta/recipes-devtools/python/python3/0001-cross-compile-support.patch
+++ b/meta/recipes-devtools/python/python3/0001-cross-compile-support.patch
@@ -34,15 +34,6 @@ index a88b7d5..7cb8bb3 100644
PSRCS= \
Parser/acceler.c \
-@@ -510,7 +512,7 @@ build_all_generate_profile:
-
- run_profile_task:
- : # FIXME: can't run for a cross build
-- $(LLVM_PROF_FILE) $(RUNSHARED) ./$(BUILDPYTHON) $(PROFILE_TASK) || true
-+ $(LLVM_PROF_FILE) $(RUNSHARED) $(HOSTPYTHON) $(PROFILE_TASK) || true
-
- build_all_merge_profile:
- $(LLVM_PROF_MERGER)
@@ -787,7 +789,7 @@ $(IO_OBJS): $(IO_H)
$(GRAMMAR_H): @GENERATED_COMMENT@ $(GRAMMAR_INPUT) $(PGEN)
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v5 02/10] python3: fix depends of python3-tests
2017-10-27 10:07 [PATCH v5 00/10] support profile-optimized build for Python Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 01/10] python3-native: support profile optimized build Markus Lehtonen
@ 2017-10-27 10:07 ` Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 03/10] python3: add python3-profile-opt recipe Markus Lehtonen
` (10 subsequent siblings)
12 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-10-27 10:07 UTC (permalink / raw)
To: openembedded-core
Similar to an earlier fix for Python 2.7. Make the tests subpackage
depend on all modules as test.regrtest uses most (if not all) of them.
Some tests also depend on libgcc so add that as a runtime dependency as
well.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python-3.5-manifest.inc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/python/python-3.5-manifest.inc b/meta/recipes-devtools/python/python-3.5-manifest.inc
index 0260e87e75..067f6c58c5 100644
--- a/meta/recipes-devtools/python/python-3.5-manifest.inc
+++ b/meta/recipes-devtools/python/python-3.5-manifest.inc
@@ -241,7 +241,7 @@ RDEPENDS_${PN}-terminal="${PN}-core ${PN}-io"
FILES_${PN}-terminal="${libdir}/python3.5/pty.* ${libdir}/python3.5/__pycache__/pty.* ${libdir}/python3.5/tty.* ${libdir}/python3.5/__pycache__/tty.* "
SUMMARY_${PN}-tests="Python tests"
-RDEPENDS_${PN}-tests="${PN}-core ${PN}-compression"
+RDEPENDS_${PN}-tests="${PN}-core ${PN}-modules libgcc"
FILES_${PN}-tests="${libdir}/python3.5/test ${libdir}/python3.5/test/__pycache__ "
SUMMARY_${PN}-textutils="Python option parsing, text wrapping and CSV support"
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v5 03/10] python3: add python3-profile-opt recipe
2017-10-27 10:07 [PATCH v5 00/10] support profile-optimized build for Python Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 01/10] python3-native: support profile optimized build Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 02/10] python3: fix depends of python3-tests Markus Lehtonen
@ 2017-10-27 10:07 ` Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 04/10] python3-profile-opt: rename libpython3 Markus Lehtonen
` (9 subsequent siblings)
12 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-10-27 10:07 UTC (permalink / raw)
To: openembedded-core
This patch adds a new recipe that builds a special version of python3 that
produces profile data used for optimization. The new recipe directly includes
the base python recipe so that they are build in as similar way as possible and
this hopefully decreases the recipe maintenance burden, too. Also, its files
clash with the "normal" python3 so they cannot be installed in an image at the
same time.
Normally, profile-guided-optimization in Python is done simply by doing "make
profile-opt" which first builds python with profile instrumentation enabled,
then runs a profile task to get the profile data, and last, re-builds python
with profile data guiding the optimization. However, in our cross-build
environment this gets a lot trickier. We need to split out the steps as
building is done on the build host but we need to run the second step (i.e. run
the profile task) on the target hardware. This patch enables the first step,
i.e. building python with profile instrumentation enabled
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python3-profile-opt | 1 +
.../python/python3-profile-opt_3.5.3.bb | 9 +++++++
...efile-add-install_generate_profile-target.patch | 25 ++++++++++++++++++
meta/recipes-devtools/python/python3_3.5.3.bb | 30 ++++++++++++----------
4 files changed, 51 insertions(+), 14 deletions(-)
create mode 120000 meta/recipes-devtools/python/python3-profile-opt
create mode 100644 meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
create mode 100644 meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch
diff --git a/meta/recipes-devtools/python/python3-profile-opt b/meta/recipes-devtools/python/python3-profile-opt
new file mode 120000
index 0000000000..b8a0adbbb9
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-profile-opt
@@ -0,0 +1 @@
+python3
\ No newline at end of file
diff --git a/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb b/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
new file mode 100644
index 0000000000..6940480546
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
@@ -0,0 +1,9 @@
+require python3_${PV}.bb
+
+SRC_URI += "file://Makefile-add-install_generate_profile-target.patch \
+ "
+
+PYTHON3_MAKE_TARGET = "build_all_generate_profile"
+
+RCONFLICTS_${PN}-core = "python3-core"
+RCONFLICTS_lib${BPN} = "libpython3"
diff --git a/meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch b/meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch
new file mode 100644
index 0000000000..7aee7fb799
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch
@@ -0,0 +1,25 @@
+Makefile: add install_generate_profile target
+
+Making it possible to install the binaries with profile generation
+enabled. We need this as linking is also performed at the install stage.
+
+Upstream-Status: Inappropriate [oe-specific]
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 2b87614..826155b 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1046,6 +1046,8 @@ QUICKTESTOPTS= $(TESTOPTS) -x test_subprocess test_io test_lib2to3 \
+ quicktest: all platform
+ $(TESTRUNNER) $(QUICKTESTOPTS)
+
++install_generate_profile:
++ $(MAKE) install LDFLAGS="$(LDFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LIBS="$(LIBS)"
+
+ install: @FRAMEWORKINSTALLFIRST@ commoninstall bininstall maninstall @FRAMEWORKINSTALLLAST@
+ if test "x$(ENSUREPIP)" != "xno" ; then \
+--
+2.6.6
+
diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
index 13df12fe56..3522b74991 100644
--- a/meta/recipes-devtools/python/python3_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3_3.5.3.bb
@@ -115,14 +115,10 @@ do_compile() {
# then call do_install twice we get Makefile.orig == Makefile.sysroot
install -m 0644 Makefile Makefile.sysroot
- oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
- HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python3-native/python3 \
- STAGING_LIBDIR=${STAGING_LIBDIR} \
- STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
- STAGING_INCDIR=${STAGING_INCDIR} \
- LIB=${baselib} \
- ARCH=${TARGET_ARCH} \
- OPT="${CFLAGS}" libpython3.so
+ if [ "${PYTHON3_MAKE_TARGET}" = "build_all_generate_profile" ]; then
+ # This is only used in PGO profiling by python3-profile-opt package
+ export EXTRA_CFLAGS="-fprofile-dir=./python3-pgo-profiles/"
+ fi
oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python3-native/python3 \
@@ -131,7 +127,7 @@ do_compile() {
STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
LIB=${baselib} \
ARCH=${TARGET_ARCH} \
- OPT="${CFLAGS}"
+ OPT="${CFLAGS}" ${PYTHON3_MAKE_TARGET}
}
do_install() {
@@ -151,8 +147,14 @@ do_install() {
STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
LIB=${baselib} \
ARCH=${TARGET_ARCH} \
- DESTDIR=${D} LIBDIR=${libdir}
+ DESTDIR=${D} LIBDIR=${libdir} ${PYTHON3_MAKE_TARGET}
+ if [ "${PYTHON3_MAKE_TARGET}" = "build_all_generate_profile" ]; then
+ # Need special make install if pgo generation is enabled
+ _PYTHON3_MAKE_INSTALL_TARGET="install_generate_profile"
+ else
+ _PYTHON3_MAKE_INSTALL_TARGET="install"
+ fi
oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python3-native/python3 \
STAGING_LIBDIR=${STAGING_LIBDIR} \
@@ -160,7 +162,7 @@ do_install() {
STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
LIB=${baselib} \
ARCH=${TARGET_ARCH} \
- DESTDIR=${D} LIBDIR=${libdir} install
+ DESTDIR=${D} LIBDIR=${libdir} ${_PYTHON3_MAKE_INSTALL_TARGET}
# avoid conflict with 2to3 from Python 2
rm -f ${D}/${bindir}/2to3
@@ -209,9 +211,9 @@ PACKAGES =+ "${PN}-pyvenv"
FILES_${PN}-pyvenv += "${bindir}/pyvenv-${PYTHON_MAJMIN} ${bindir}/pyvenv"
# package libpython3
-PACKAGES =+ "libpython3 libpython3-staticdev"
-FILES_libpython3 = "${libdir}/libpython*.so.*"
-FILES_libpython3-staticdev += "${libdir}/python${PYTHON_MAJMIN}/config-${PYTHON_BINABI}/libpython${PYTHON_BINABI}.a"
+PACKAGES =+ "lib${BPN} lib${BPN}-staticdev"
+FILES_lib${BPN} = "${libdir}/libpython*.so.*"
+FILES_lib${BPN}-staticdev += "${libdir}/python${PYTHON_MAJMIN}/config-${PYTHON_BINABI}/libpython*.a"
INSANE_SKIP_${PN}-dev += "dev-elf"
# catch all the rest (unsorted)
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v5 04/10] python3-profile-opt: rename libpython3
2017-10-27 10:07 [PATCH v5 00/10] support profile-optimized build for Python Markus Lehtonen
` (2 preceding siblings ...)
2017-10-27 10:07 ` [PATCH v5 03/10] python3: add python3-profile-opt recipe Markus Lehtonen
@ 2017-10-27 10:07 ` Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 05/10] bitbake.conf: add ip and scp to HOSTTOOLS unconditionally Markus Lehtonen
` (8 subsequent siblings)
12 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-10-27 10:07 UTC (permalink / raw)
To: openembedded-core
Prevents soname clash with "normal" libpython3. The python library
needs to be renamed so that the automatic ndency generation/checking
mechanism in bitbake does not get confused. Otherwise
python3-profile-opt will depend on libpython3 from the "normal" python3
package.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
.../python/python3-profile-opt_3.5.3.bb | 1 +
...name-libpython3-to-libpython-profile-opt3.patch | 108 +++++++++++++++++++++
2 files changed, 109 insertions(+)
create mode 100644 meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch
diff --git a/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb b/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
index 6940480546..3ad427117f 100644
--- a/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
@@ -1,6 +1,7 @@
require python3_${PV}.bb
SRC_URI += "file://Makefile-add-install_generate_profile-target.patch \
+ file://rename-libpython3-to-libpython-profile-opt3.patch \
"
PYTHON3_MAKE_TARGET = "build_all_generate_profile"
diff --git a/meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch b/meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch
new file mode 100644
index 0000000000..2e5f9302fc
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch
@@ -0,0 +1,108 @@
+rename libpython3 to libpython-profile-opt3
+
+Upstream-Status: Inappropriate [oe-specific]
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+
+diff --git a/Lib/distutils/command/build_ext.py b/Lib/distutils/command/build_ext.py
+index 7c278ef..78613d8 100644
+--- a/Lib/distutils/command/build_ext.py
++++ b/Lib/distutils/command/build_ext.py
+@@ -747,7 +747,7 @@ class build_ext(Command):
+ else:
+ from distutils import sysconfig
+ if sysconfig.get_config_var('Py_ENABLE_SHARED'):
+- pythonlib = 'python{}.{}{}'.format(
++ pythonlib = 'python-profile-opt{}.{}{}'.format(
+ sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff,
+ sysconfig.get_config_var('ABIFLAGS'))
+ return ext.libraries + [pythonlib]
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 0327385..f744ce1 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -623,7 +623,7 @@ $(LIBRARY): $(LIBRARY_OBJS)
+ $(AR) $(ARFLAGS) $@ $(MODOBJS)
+ $(RANLIB) $@
+
+-libpython$(LDVERSION).so: $(LIBRARY_OBJS)
++libpython-profile-opt$(LDVERSION).so: $(LIBRARY_OBJS)
+ if test $(INSTSONAME) != $(LDLIBRARY); then \
+ $(BLDSHARED) -Wl,-h$(INSTSONAME) -o $(INSTSONAME) $(LIBRARY_OBJS) $(MODLIBS) $(SHLIBS) $(LIBC) $(LIBM) $(LDLAST); \
+ $(LN) -f $(INSTSONAME) $@; \
+@@ -631,7 +631,7 @@ libpython$(LDVERSION).so: $(LIBRARY_OBJS)
+ $(BLDSHARED) -o $@ $(LIBRARY_OBJS) $(MODLIBS) $(SHLIBS) $(LIBC) $(LIBM) $(LDLAST); \
+ fi
+
+-libpython3.so: libpython$(LDVERSION).so
++libpython-profile-opt3.so: libpython-profile-opt$(LDVERSION).so
+ $(BLDSHARED) $(NO_AS_NEEDED) -o $@ -Wl,-h$@ $^
+
+ libpython$(LDVERSION).dylib: $(LIBRARY_OBJS)
+diff --git a/Misc/python-config.in b/Misc/python-config.in
+index a808f1a..0dcc615 100644
+--- a/Misc/python-config.in
++++ b/Misc/python-config.in
+@@ -47,7 +47,7 @@ for opt in opt_flags:
+ print(' '.join(flags))
+
+ elif opt in ('--libs', '--ldflags'):
+- libs = ['-lpython' + pyver + sys.abiflags]
++ libs = ['-lpython-profile-opt' + pyver + sys.abiflags]
+ libs += getvar('LIBS').split()
+ libs += getvar('SYSLIBS').split()
+ # add the prefix/lib/pythonX.Y/config dir, but only if there is no
+diff --git a/Misc/python-config.sh.in b/Misc/python-config.sh.in
+index 30c6927..efad748 100644
+--- a/Misc/python-config.sh.in
++++ b/Misc/python-config.sh.in
+@@ -40,7 +40,7 @@ LIBM="@LIBM@"
+ LIBC="@LIBC@"
+ SYSLIBS="$LIBM $LIBC"
+ ABIFLAGS="@ABIFLAGS@"
+-LIBS="-lpython${VERSION}${ABIFLAGS} @LIBS@ $SYSLIBS"
++LIBS="-lpython-profile-opt${VERSION}${ABIFLAGS} @LIBS@ $SYSLIBS"
+ BASECFLAGS="@BASECFLAGS@"
+ LDLIBRARY="@LDLIBRARY@"
+ LINKFORSHARED="@LINKFORSHARED@"
+diff --git a/Misc/python.pc.in b/Misc/python.pc.in
+index ae69867..b73d027 100644
+--- a/Misc/python.pc.in
++++ b/Misc/python.pc.in
+@@ -9,5 +9,5 @@ Description: Python library
+ Requires:
+ Version: @VERSION@
+ Libs.private: @LIBS@
+-Libs: -L${libdir} -lpython@VERSION@@ABIFLAGS@
++Libs: -L${libdir} -lpython-profile-opt@VERSION@@ABIFLAGS@
+ Cflags: -I${includedir}/python@VERSION@@ABIFLAGS@
+diff --git a/configure.ac b/configure.ac
+index d8c2847..ad63e4e 100644
+--- a/configure.ac
++++ b/configure.ac
+@@ -1001,7 +1001,7 @@ AC_SUBST(LIBRARY)
+ AC_MSG_CHECKING(LIBRARY)
+ if test -z "$LIBRARY"
+ then
+- LIBRARY='libpython$(VERSION)$(ABIFLAGS).a'
++ LIBRARY='libpython-profile-opt$(VERSION)$(ABIFLAGS).a'
+ fi
+ AC_MSG_RESULT($LIBRARY)
+
+@@ -1162,13 +1162,13 @@ if test $enable_shared = "yes"; then
+ fi
+ ;;
+ Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*)
+- LDLIBRARY='libpython$(LDVERSION).so'
+- BLDLIBRARY='-L. -lpython$(LDVERSION)'
++ LDLIBRARY='libpython-profile-opt$(LDVERSION).so'
++ BLDLIBRARY='-L. -lpython-profile-opt$(LDVERSION)'
+ RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
+ INSTSONAME="$LDLIBRARY".$SOVERSION
+ if test "$with_pydebug" != yes
+ then
+- PY3LIBRARY=libpython3.so
++ PY3LIBRARY=libpython-profile-opt3.so
+ fi
+ ;;
+ hp*|HP*)
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v5 05/10] bitbake.conf: add ip and scp to HOSTTOOLS unconditionally
2017-10-27 10:07 [PATCH v5 00/10] support profile-optimized build for Python Markus Lehtonen
` (3 preceding siblings ...)
2017-10-27 10:07 ` [PATCH v5 04/10] python3-profile-opt: rename libpython3 Markus Lehtonen
@ 2017-10-27 10:07 ` Markus Lehtonen
2017-10-30 1:30 ` ChenQi
2017-10-27 10:07 ` [PATCH v5 06/10] devtools/images: add python-pgo-image Markus Lehtonen
` (7 subsequent siblings)
12 siblings, 1 reply; 41+ messages in thread
From: Markus Lehtonen @ 2017-10-27 10:07 UTC (permalink / raw)
To: openembedded-core
Needed to be able to run images using the runtime test facilities.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/conf/bitbake.conf | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 9696273acc..9075d03998 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -476,15 +476,15 @@ HOSTTOOLS += " \
[ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp cp cpio \
cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
- head hostname id install ld ldd ln ls make makeinfo md5sum mkdir mknod \
+ head hostname id install ip ld ldd ln ls make makeinfo md5sum mkdir mknod \
mktemp mv nm objcopy objdump od patch perl pod2man pr printf pwd python python2 \
- python2.7 python3 ranlib readelf readlink rm rmdir rpcgen sed sh sha256sum \
+ python2.7 python3 ranlib readelf readlink rm rmdir rpcgen scp sed sh sha256sum \
sleep sort split stat strings strip tail tar tee test touch tr true uname \
uniq wc wget which xargs \
"
# Tools needed to run testimage runtime image testing
-HOSTTOOLS += "${@['', 'ip ping ps scp ssh stty'][bb.data.inherits_class('testimage', d)]}"
+HOSTTOOLS += "${@['', 'ping ps ssh stty'][bb.data.inherits_class('testimage', d)]}"
# Link to these if present
HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc sftp socat ssh sudo"
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v5 06/10] devtools/images: add python-pgo-image
2017-10-27 10:07 [PATCH v5 00/10] support profile-optimized build for Python Markus Lehtonen
` (4 preceding siblings ...)
2017-10-27 10:07 ` [PATCH v5 05/10] bitbake.conf: add ip and scp to HOSTTOOLS unconditionally Markus Lehtonen
@ 2017-10-27 10:07 ` Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 07/10] python3: support profile optimized build Markus Lehtonen
` (6 subsequent siblings)
12 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-10-27 10:07 UTC (permalink / raw)
To: openembedded-core
This is a special image for profiling Python 3.x in order to utilize
profile-guided-optimization. Profile data can be obtained by by running
bitbake python-pgo-image -c profile. Profile data will be copied into a
directory pointed by PYTHON3_PROFILE_DIR on the host system. The profile
task may be specified with PYTHON3_PROFILE_TASK.
The default profile task is pybench, although an example for using
test.regrtest is included in the recipe.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/images/python-pgo-image.bb | 92 ++++++++++++++++++++++++
1 file changed, 92 insertions(+)
create mode 100644 meta/recipes-devtools/images/python-pgo-image.bb
diff --git a/meta/recipes-devtools/images/python-pgo-image.bb b/meta/recipes-devtools/images/python-pgo-image.bb
new file mode 100644
index 0000000000..6ba5f09389
--- /dev/null
+++ b/meta/recipes-devtools/images/python-pgo-image.bb
@@ -0,0 +1,92 @@
+SUMMARY = "Minimal image for doing Python profiling (for PGO)"
+
+IMAGE_FEATURES += "ssh-server-dropbear"
+IMAGE_INSTALL = "packagegroup-core-boot"
+IMAGE_INSTALL += "python3-profile-opt python3-profile-opt-tests python3-profile-opt-tools"
+
+LICENSE = "MIT"
+
+inherit core-image
+
+PROFILE_DATA_WORKDIR = "${WORKDIR}/profile-data"
+
+PYTHON3_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python3/pgo-data"
+PYTHON3_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_main_handling test_multiprocessing_spawn test_subprocess"
+# Exclude tests that are failing on qemux86
+PYTHON3_PROFILE_TASK_DEFAULT += "test_builtin test_cmath test_concurrent_futures test_difflib test_distutils test_float test_format test_math test_optparse test_shutil test_statistics test_types test_unicode"
+# Exclude tests that are taking very long on qemux86
+PYTHON3_PROFILE_TASK_DEFAULT += "test_lib2to3 test_buffer test_pickle test_io test_threading test_asyncio test_urllib2_localnet test_itertools test_tuple test_trace test_tarfile test_unicodedata test_decimal test_long test_zipfile test_deque test_descr test_email test_venv test_bytes test_compileall test_ast test_multibytecodec"
+
+# Change default profile target to pybench. Running test.regrtest takes
+# ridiculously long, i.e. around 4 hours in qemux86 on i7-3770K desktop
+# machine. Pybench "only" takes around 55 minutes.
+PYTHON3_PROFILE_TASK_DEFAULT = "${docdir}/python3-profile-opt/Tools/pybench/pybench.py -n 2 --with-gc --with-syscheck"
+
+PYTHON3_PROFILE_TASK ?= "${PYTHON3_PROFILE_TASK_DEFAULT}"
+
+# We need these because we're utilizing the runtime test helpers from oeqa
+TEST_TARGET ?= "qemu"
+TEST_QEMUBOOT_TIMEOUT ?= "1000"
+TEST_LOG_DIR ?= "${WORKDIR}/qemulogs"
+FIND_ROOTFS = "1"
+
+def run_profile(d, profile_bin, profile_task, tgt_in_dir, host_out_dir):
+ from oeqa.targetcontrol import get_target_controller
+ from oe.utils import getstatusoutput
+
+ target = get_target_controller(d)
+ target.deploy()
+ try:
+ # Boot target
+ bootparams = None
+ if d.getVar('VIRTUAL-RUNTIME_init_manager', True) == 'systemd':
+ bootparams = 'systemd.log_level=debug systemd.log_target=console'
+ target.start(extra_bootparams=bootparams)
+
+ # Run profile task
+ ret, output = target.run(profile_bin + ' ' + profile_task, timeout=7200)
+ if ret:
+ bb.fatal("Failed to run profile task on target: %s" % output)
+ ret, output = target.run('tar czf pgo-data.tgz -C %s .' % tgt_in_dir)
+ if ret:
+ bb.fatal("Failed to archive profile data on target: %s" % output)
+
+ # Retrieve and unpack profile data
+ target.copy_from('/home/root/pgo-data.tgz', host_out_dir)
+
+ profile_tarball = os.path.join(host_out_dir, 'pgo-data.tgz')
+ ret, output = getstatusoutput('tar xf %s -C %s' % (profile_tarball, host_out_dir))
+ os.unlink(profile_tarball)
+ if ret:
+ bb.fatal("Failed to unpack python profile data: %s" % output)
+ finally:
+ target.stop()
+
+
+# Profile task for Python 3.x
+python do_profile() {
+ run_profile(d, 'python3',
+ d.getVar('PYTHON3_PROFILE_TASK', True),
+ '/home/root/python3-pgo-profiles/',
+ os.path.join(d.getVar("PROFILE_DATA_WORKDIR", True), 'python3'))
+}
+
+# Awkward dependency trick: add 'do_profile' task after 'do_build', as
+# expected, so that everything (image and kernel) is in place. However, add
+# the 'populate_sysroot' tasks as dependencies of 'do_image_complete' (instead
+# of 'do_build') in order to avoid all possible build deps to be installed in
+# the recipe sysroot. This would cause both python3 and python3-profile-opt
+# to be installed which clash (duplicate files).
+#
+addtask profile after do_build
+do_image_complete[depends] += "qemu-native:do_populate_sysroot qemu-helper-native:do_populate_sysroot"
+do_profile[cleandirs] = "${PROFILE_DATA_WORKDIR}/python3"
+
+python do_profile_setscene () {
+ sstate_setscene(d)
+}
+
+SSTATETASKS += "do_profile"
+do_profile[sstate-inputdirs] = "${PROFILE_DATA_WORKDIR}/python3"
+do_profile[sstate-outputdirs] = "${PYTHON3_PROFILE_DIR}"
+addtask do_profile_setscene
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v5 07/10] python3: support profile optimized build
2017-10-27 10:07 [PATCH v5 00/10] support profile-optimized build for Python Markus Lehtonen
` (5 preceding siblings ...)
2017-10-27 10:07 ` [PATCH v5 06/10] devtools/images: add python-pgo-image Markus Lehtonen
@ 2017-10-27 10:07 ` Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 08/10] python3: fix profile-optimized build of modules Markus Lehtonen
` (5 subsequent siblings)
12 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-10-27 10:07 UTC (permalink / raw)
To: openembedded-core
This patch makes it possible to build python3 with profile directed
optimization. That is, feed python build process with profile data to
guide optimization. This is the third (and the last) step in profile
directed optimization for Python 3.x.
In order to do a profile-optimized build you need to specify
PYTHON3_PROFILE_OPT = "1" in your local.conf, and, have profile data available
in the location pointed to by PYTHON3_PROFILE_DIR. Profile data can be
obtainen e.g. by running bitbake python-pgo-image -c profile3.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python3_3.5.3.bb | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
index 3522b74991..1115c3feba 100644
--- a/meta/recipes-devtools/python/python3_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3_3.5.3.bb
@@ -79,6 +79,17 @@ export CROSSPYTHONPATH = "${STAGING_LIBDIR_NATIVE}/python${PYTHON_MAJMIN}/lib-dy
# No ctypes option for python 3
PYTHONLSBOPTS = ""
+# Automatic profile guided optimization
+PYTHON3_MAKE_TARGET ?= "${@'build_all_use_profile' if d.getVar('PYTHON3_PROFILE_OPT', True) == '1' else ''}"
+PYTHON3_PROFILE_DIR ?= "${@'${TMPDIR}/work-shared/${MACHINE}/python3/pgo-data' if d.getVar('PYTHON3_PROFILE_OPT', True) == '1' else ''}"
+python () {
+ if (d.getVar('PYTHON3_PROFILE_OPT', True) == '1' and
+ d.getVar('PYTHON3_MAKE_TARGET', True) == 'build_all_use_profile'):
+ profile_dir = d.getVar('PYTHON3_PROFILE_DIR', True)
+ bb.utils.mkdirhier(profile_dir)
+ d.setVarFlag('do_compile', 'file-checksums', '%s:True' % profile_dir)
+}
+
do_configure_append() {
rm -f ${S}/Makefile.orig
autoreconf -Wcross --verbose --install --force --exclude=autopoint ../Python-${PV}/Modules/_ctypes/libffi
@@ -118,6 +129,10 @@ do_compile() {
if [ "${PYTHON3_MAKE_TARGET}" = "build_all_generate_profile" ]; then
# This is only used in PGO profiling by python3-profile-opt package
export EXTRA_CFLAGS="-fprofile-dir=./python3-pgo-profiles/"
+ elif [ -n "${PYTHON3_PROFILE_DIR}" ]; then
+ export EXTRA_CFLAGS="-fprofile-dir=${PYTHON3_PROFILE_DIR}"
+ # Remove non-optimized build artefacts
+ oe_runmake clean
fi
oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
@@ -138,6 +153,11 @@ do_install() {
install -d ${D}${libdir}/pkgconfig
install -d ${D}${libdir}/python${PYTHON_MAJMIN}/config
+ # This only has effect if we build with -fprofile-use, e.g. when make
+ # target is build_all_use_profile
+ if [ -n "${PYTHON3_PROFILE_DIR}" ]; then
+ export EXTRA_CFLAGS="-fprofile-dir=${PYTHON3_PROFILE_DIR}"
+ fi
# rerun the build once again with original makefile this time
# run install in a separate step to avoid compile/install race
oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v5 08/10] python3: fix profile-optimized build of modules
2017-10-27 10:07 [PATCH v5 00/10] support profile-optimized build for Python Markus Lehtonen
` (6 preceding siblings ...)
2017-10-27 10:07 ` [PATCH v5 07/10] python3: support profile optimized build Markus Lehtonen
@ 2017-10-27 10:07 ` Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 09/10] python3: add python3-tools subpackage Markus Lehtonen
` (4 subsequent siblings)
12 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-10-27 10:07 UTC (permalink / raw)
To: openembedded-core
Without this the pgo-related compiler flags are not used in
cross-builds.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
...-CFLAGS-for-extensions-when-cross-compili.patch | 56 ++++++++++++++++++++++
meta/recipes-devtools/python/python3_3.5.3.bb | 1 +
2 files changed, 57 insertions(+)
create mode 100644 meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch
diff --git a/meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch b/meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch
new file mode 100644
index 0000000000..2e299b3c59
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch
@@ -0,0 +1,56 @@
+From d721ebabbf8567d42dfdcd17e170846ba837b04a Mon Sep 17 00:00:00 2001
+From: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+Date: Wed, 23 Nov 2016 16:08:04 +0200
+Subject: [PATCH] Use correct CFLAGS for extensions when cross-compiling
+
+Take PY_CFLAGS_NODIST into account, like in native build. This is needed
+in order to to profile-optimized build. Also, pass EXTRA_CFLAGS to
+profile-optimized build.
+
+Upstream-Status: Pending
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+---
+ Makefile.pre.in | 4 ++--
+ setup.py | 3 ++-
+ 2 files changed, 4 insertions(+), 3 deletions(-)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 4fcaafc..2d20753 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -510,7 +510,7 @@ profile-opt:
+ $(MAKE) profile-removal
+
+ build_all_generate_profile:
+- $(MAKE) @DEF_MAKE_RULE@ CFLAGS_NODIST="$(CFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LIBS="$(LIBS)"
++ $(MAKE) @DEF_MAKE_RULE@ CFLAGS_NODIST="$(CFLAGS) $(EXTRA_CFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LIBS="$(LIBS)"
+
+ run_profile_task:
+ : # FIXME: can't run for a cross build
+@@ -520,7 +520,7 @@ build_all_merge_profile:
+ $(LLVM_PROF_MERGER)
+
+ build_all_use_profile:
+- $(MAKE) @DEF_MAKE_RULE@ CFLAGS_NODIST="$(CFLAGS) $(PGO_PROF_USE_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) @LTOFLAGS@"
++ $(MAKE) @DEF_MAKE_RULE@ CFLAGS_NODIST="$(CFLAGS) $(EXTRA_CFLAGS) $(PGO_PROF_USE_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) @LTOFLAGS@"
+
+ # Compile and run with gcov
+ .PHONY=coverage coverage-lcov coverage-report
+diff --git a/setup.py b/setup.py
+index 7fa9cc3..ef18fe4 100644
+--- a/setup.py
++++ b/setup.py
+@@ -263,7 +263,8 @@ class PyBuildExt(build_ext):
+ # compilers
+ if compiler is not None:
+ if cross_compiling:
+- (ccshared,cflags) = (os.environ.get('CCSHARED') or '', os.environ.get('CFLAGS') or '')
++ (ccshared,cflags) = (os.environ.get('CCSHARED') or '',
++ (os.environ.get('CFLAGS') or '') + ' ' + sysconfig.get_config_var('PY_CFLAGS_NODIST'))
+ else:
+ (ccshared,cflags) = sysconfig.get_config_vars('CCSHARED','CFLAGS')
+ args['compiler_so'] = compiler + ' ' + ccshared + ' ' + cflags
+--
+2.12.3
+
diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
index 1115c3feba..b8d89b5979 100644
--- a/meta/recipes-devtools/python/python3_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3_3.5.3.bb
@@ -40,6 +40,7 @@ SRC_URI += "\
file://0001-Issue-21272-Use-_sysconfigdata.py-to-initialize-dist.patch \
file://Fix-29519-weakref-spewing-exceptions-during-interp-f.patch \
file://pass-missing-libraries-to-Extension-for-mul.patch \
+ file://Use-correct-CFLAGS-for-extensions-when-cross-compili.patch \
"
SRC_URI[md5sum] = "57d1f8bfbabf4f2500273fb0706e6f21"
SRC_URI[sha256sum] = "eefe2ad6575855423ab630f5b51a8ef6e5556f774584c06beab4926f930ddbb0"
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v5 09/10] python3: add python3-tools subpackage
2017-10-27 10:07 [PATCH v5 00/10] support profile-optimized build for Python Markus Lehtonen
` (7 preceding siblings ...)
2017-10-27 10:07 ` [PATCH v5 08/10] python3: fix profile-optimized build of modules Markus Lehtonen
@ 2017-10-27 10:07 ` Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 10/10] python3: remove two setup.py cross-compile hacks Markus Lehtonen
` (3 subsequent siblings)
12 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-10-27 10:07 UTC (permalink / raw)
To: openembedded-core
Useful in developing Python, e.g. in benchmarking.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python-3.5-manifest.inc | 8 +++++---
meta/recipes-devtools/python/python3_3.5.3.bb | 5 +++++
2 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/meta/recipes-devtools/python/python-3.5-manifest.inc b/meta/recipes-devtools/python/python-3.5-manifest.inc
index 067f6c58c5..3107d48f7e 100644
--- a/meta/recipes-devtools/python/python-3.5-manifest.inc
+++ b/meta/recipes-devtools/python/python-3.5-manifest.inc
@@ -4,9 +4,9 @@
-PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc "
+PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-tools "
-PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-modules"
+PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-modules ${PN}-tools"
SUMMARY_${PN}-2to3="Python automated Python 2 to 3 code translator"
RDEPENDS_${PN}-2to3="${PN}-core"
@@ -280,4 +280,6 @@ SUMMARY_${PN}-modules="All Python modules"
RDEPENDS_${PN}-modules="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-difflib ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc "
ALLOW_EMPTY_${PN}-modules = "1"
-
+SUMMARY_${PN}-tools="Useful scripts from Python"
+RDEPENDS_${PN}-tools="${PN}-modules"
+FILES_${PN}-tools="${docdir}/${PN}/Tools"
diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
index b8d89b5979..b60bfcdb12 100644
--- a/meta/recipes-devtools/python/python3_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3_3.5.3.bb
@@ -196,6 +196,11 @@ do_install() {
fi
oe_multilib_header python${PYTHON_BINABI}/pyconfig.h
+
+ # Install tools
+ install -d -m 755 ${D}/${docdir}/${PN}
+ cp -r ${S}/Tools ${D}/${docdir}/${PN}/
+ find ${D}/${docdir}/${PN}/ -name Makefile -delete
}
do_install_append_class-nativesdk () {
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v5 10/10] python3: remove two setup.py cross-compile hacks
2017-10-27 10:07 [PATCH v5 00/10] support profile-optimized build for Python Markus Lehtonen
` (8 preceding siblings ...)
2017-10-27 10:07 ` [PATCH v5 09/10] python3: add python3-tools subpackage Markus Lehtonen
@ 2017-10-27 10:07 ` Markus Lehtonen
2017-10-27 10:34 ` ✗ patchtest: failure for support profile-optimized build for Python Patchwork
` (2 subsequent siblings)
12 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-10-27 10:07 UTC (permalink / raw)
To: openembedded-core
Remove two unneeded hacks. The first hack ("setup.py: no host headers
libs" patch) is not needed because we use cross-compiler (e.g.
i586-oe-linux-gcc) which has not been configured with any host system
include or library directories, and thus, we don't get any host system
directories when running "gcc -E -v".
The second hack becomes useless after the first hack has been removed
and we get the standard include and lib directories normally from gcc.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
| 33 ----------------------
.../setup.py-find-libraries-in-staging-dirs.patch | 30 --------------------
meta/recipes-devtools/python/python3_3.5.3.bb | 2 --
3 files changed, 65 deletions(-)
delete mode 100644 meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch
delete mode 100644 meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
diff --git a/meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch b/meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch
deleted file mode 100644
index 2bc8b8c460..0000000000
--- a/meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch
+++ /dev/null
@@ -1,33 +0,0 @@
-From 43238e1ac13e32984d015c92a5841f3de1fe1d15 Mon Sep 17 00:00:00 2001
-From: Jackie Huang <jackie.huang@windriver.com>
-Date: Tue, 18 Nov 2014 00:07:07 -0500
-Subject: [PATCH] setup.py: no host headers libs
-
-When we are cross-compiling, setup.py should never look in /usr
-or /usr/local to find headers or libraries.
-
-Upstream-Status: Inappropriate [Cross compile specific]
-
-Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
----
- setup.py | 3 ---
- 1 file changed, 3 deletions(-)
-
-diff --git a/setup.py b/setup.py
-index f020b28..e8339cd 100644
---- a/setup.py
-+++ b/setup.py
-@@ -444,10 +444,7 @@ class PyBuildExt(build_ext):
- if not cross_compiling:
- add_dir_to_list(self.compiler.library_dirs, os.path.join('/usr/local', sys.lib))
- add_dir_to_list(self.compiler.include_dirs, '/usr/local/include')
-- # only change this for cross builds for 3.3, issues on Mageia
-- if cross_compiling:
- self.add_gcc_paths()
-- if not cross_compiling:
- self.add_multiarch_paths()
-
- # Add paths specified in the environment variables LDFLAGS and
---
-2.0.0
-
diff --git a/meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch b/meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
deleted file mode 100644
index f26bd38b02..0000000000
--- a/meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
+++ /dev/null
@@ -1,30 +0,0 @@
-Patch setup.py so that the detect_modules() function looks for required
-libraries and headers in STAGING_LIBDIR / STAGING_INCDIR.
-
-Without this patch, several extension modules are not built, even though
-their dependencies are present in the compiler's search paths.
-The result is the following warning, and ultimately incomplete packages:
-
-| The necessary bits to build these optional modules were not found:
-| _bz2 _curses_panel _dbm
-| _gdbm _lzma _sqlite3
-| nis readline zlib
-| To find the necessary bits, look in setup.py in detect_modules() for the module's name.
-
-Upstream-Status: Inappropriate [Cross compile specific]
-
-Signed-off-by: Dominic Sacré <dominic.sacre@gmx.de>
-
-Index: Python-3.4.3/setup.py
-===================================================================
---- Python-3.4.3.orig/setup.py
-+++ Python-3.4.3/setup.py
-@@ -521,6 +521,8 @@ class PyBuildExt(build_ext):
- else:
- lib_dirs = self.compiler.library_dirs[:]
- inc_dirs = self.compiler.include_dirs[:]
-+ lib_dirs.append(os.environ.get('STAGING_LIBDIR'))
-+ inc_dirs.append(os.environ.get('STAGING_INCDIR'))
- exts = []
- missing = []
-
diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
index b60bfcdb12..41d0e48d42 100644
--- a/meta/recipes-devtools/python/python3_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3_3.5.3.bb
@@ -31,10 +31,8 @@ SRC_URI += "\
file://unixccompiler.patch \
file://avoid-ncursesw-include-path.patch \
file://python3-use-CROSSPYTHONPATH-for-PYTHON_FOR_BUILD.patch \
- file://python3-setup.py-no-host-headers-libs.patch \
file://sysconfig.py-add-_PYTHON_PROJECT_SRC.patch \
file://setup.py-check-cross_compiling-when-get-FLAGS.patch \
- file://setup.py-find-libraries-in-staging-dirs.patch \
file://configure.ac-fix-LIBPL.patch \
file://upstream-random-fixes.patch \
file://0001-Issue-21272-Use-_sysconfigdata.py-to-initialize-dist.patch \
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* ✗ patchtest: failure for support profile-optimized build for Python
2017-10-27 10:07 [PATCH v5 00/10] support profile-optimized build for Python Markus Lehtonen
` (9 preceding siblings ...)
2017-10-27 10:07 ` [PATCH v5 10/10] python3: remove two setup.py cross-compile hacks Markus Lehtonen
@ 2017-10-27 10:34 ` Patchwork
2017-10-27 15:55 ` Daniela Plascencia
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
2017-11-20 12:33 ` ✗ patchtest: failure for support profile-optimized build for Python (rev2) Patchwork
12 siblings, 1 reply; 41+ messages in thread
From: Patchwork @ 2017-10-27 10:34 UTC (permalink / raw)
To: Markus Lehtonen; +Cc: openembedded-core
== Series Details ==
Series: support profile-optimized build for Python
Revision: 1
URL : https://patchwork.openembedded.org/series/9511/
State : failure
== Summary ==
Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:
* Issue LIC_FILES_CHKSUM changed on target python3-profile-opt but there was no explanation as to why in the commit message [test_lic_files_chksum_modified_not_mentioned]
Suggested fix Provide a reason for LIC_FILES_CHKSUM change in commit message
Current checksum
New checksum file://LICENSE;md5=b680ed99aa60d350c65a65914494207e
* Issue LIC_FILES_CHKSUM is missing in newly added recipe [test_lic_files_chksum_presence]
Suggested fix Specify the variable LIC_FILES_CHKSUM in python-pgo-image
If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).
---
Guidelines: https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: ✗ patchtest: failure for support profile-optimized build for Python
2017-10-27 10:34 ` ✗ patchtest: failure for support profile-optimized build for Python Patchwork
@ 2017-10-27 15:55 ` Daniela Plascencia
0 siblings, 0 replies; 41+ messages in thread
From: Daniela Plascencia @ 2017-10-27 15:55 UTC (permalink / raw)
To: openembedded-core, Markus Lehtonen
Please, ignore the patchtest failure message. This is a bad check and
the issue is being tracked.
On 10/27/2017 05:34 AM, Patchwork wrote:
> == Series Details ==
>
> Series: support profile-optimized build for Python
> Revision: 1
> URL : https://patchwork.openembedded.org/series/9511/
> State : failure
>
> == Summary ==
>
>
> Thank you for submitting this patch series to OpenEmbedded Core. This is
> an automated response. Several tests have been executed on the proposed
> series by patchtest resulting in the following failures:
>
>
>
> * Issue LIC_FILES_CHKSUM changed on target python3-profile-opt but there was no explanation as to why in the commit message [test_lic_files_chksum_modified_not_mentioned]
> Suggested fix Provide a reason for LIC_FILES_CHKSUM change in commit message
> Current checksum
> New checksum file://LICENSE;md5=b680ed99aa60d350c65a65914494207e
>
> * Issue LIC_FILES_CHKSUM is missing in newly added recipe [test_lic_files_chksum_presence]
> Suggested fix Specify the variable LIC_FILES_CHKSUM in python-pgo-image
>
>
>
> If you believe any of these test results are incorrect, please reply to the
> mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
> Otherwise we would appreciate you correcting the issues and submitting a new
> version of the patchset if applicable. Please ensure you add/increment the
> version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
> [PATCH v3] -> ...).
>
> ---
> Guidelines: https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
> Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
> Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
>
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v5 05/10] bitbake.conf: add ip and scp to HOSTTOOLS unconditionally
2017-10-27 10:07 ` [PATCH v5 05/10] bitbake.conf: add ip and scp to HOSTTOOLS unconditionally Markus Lehtonen
@ 2017-10-30 1:30 ` ChenQi
2017-10-30 10:51 ` Markus Lehtonen
0 siblings, 1 reply; 41+ messages in thread
From: ChenQi @ 2017-10-30 1:30 UTC (permalink / raw)
To: Markus Lehtonen, openembedded-core
On 10/27/2017 06:07 PM, Markus Lehtonen wrote:
> Needed to be able to run images using the runtime test facilities.
Hi Markus,
What do you mean by 'run images using runtime test facilities'?
testimage? ptest?
'scp' and 'ip' are added to HOSTTOOLS if 'testimage' is inherited.
What's the use case that still needs 'scp' and 'ip' unconditionally?
In addition, if we add these two utilities to HOSTTOOLS unconditionally.
Please cc to doc people to also change the QuickStart.
Regards,
Chen Qi
> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
> ---
> meta/conf/bitbake.conf | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 9696273acc..9075d03998 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -476,15 +476,15 @@ HOSTTOOLS += " \
> [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp cp cpio \
> cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
> fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
> - head hostname id install ld ldd ln ls make makeinfo md5sum mkdir mknod \
> + head hostname id install ip ld ldd ln ls make makeinfo md5sum mkdir mknod \
> mktemp mv nm objcopy objdump od patch perl pod2man pr printf pwd python python2 \
> - python2.7 python3 ranlib readelf readlink rm rmdir rpcgen sed sh sha256sum \
> + python2.7 python3 ranlib readelf readlink rm rmdir rpcgen scp sed sh sha256sum \
> sleep sort split stat strings strip tail tar tee test touch tr true uname \
> uniq wc wget which xargs \
> "
>
> # Tools needed to run testimage runtime image testing
> -HOSTTOOLS += "${@['', 'ip ping ps scp ssh stty'][bb.data.inherits_class('testimage', d)]}"
> +HOSTTOOLS += "${@['', 'ping ps ssh stty'][bb.data.inherits_class('testimage', d)]}"
>
> # Link to these if present
> HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc sftp socat ssh sudo"
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v5 05/10] bitbake.conf: add ip and scp to HOSTTOOLS unconditionally
2017-10-30 1:30 ` ChenQi
@ 2017-10-30 10:51 ` Markus Lehtonen
2017-10-30 13:55 ` Konrad Scherer
0 siblings, 1 reply; 41+ messages in thread
From: Markus Lehtonen @ 2017-10-30 10:51 UTC (permalink / raw)
To: ChenQi, openembedded-core
Hi,
What that tries state is that the image uses facilities written for runtimetest (i.e. oeqa.targetcontrol) to run binaries on the target machine. The new image introduced (i.e. python-pgo-image) uses this to run the profiling task on device.
Thanks,
Markus
On 30/10/2017, 3.29, "ChenQi" <Qi.Chen@windriver.com> wrote:
On 10/27/2017 06:07 PM, Markus Lehtonen wrote:
> Needed to be able to run images using the runtime test facilities.
Hi Markus,
What do you mean by 'run images using runtime test facilities'?
testimage? ptest?
'scp' and 'ip' are added to HOSTTOOLS if 'testimage' is inherited.
What's the use case that still needs 'scp' and 'ip' unconditionally?
In addition, if we add these two utilities to HOSTTOOLS unconditionally.
Please cc to doc people to also change the QuickStart.
Regards,
Chen Qi
> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
> ---
> meta/conf/bitbake.conf | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 9696273acc..9075d03998 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -476,15 +476,15 @@ HOSTTOOLS += " \
> [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp cp cpio \
> cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
> fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
> - head hostname id install ld ldd ln ls make makeinfo md5sum mkdir mknod \
> + head hostname id install ip ld ldd ln ls make makeinfo md5sum mkdir mknod \
> mktemp mv nm objcopy objdump od patch perl pod2man pr printf pwd python python2 \
> - python2.7 python3 ranlib readelf readlink rm rmdir rpcgen sed sh sha256sum \
> + python2.7 python3 ranlib readelf readlink rm rmdir rpcgen scp sed sh sha256sum \
> sleep sort split stat strings strip tail tar tee test touch tr true uname \
> uniq wc wget which xargs \
> "
>
> # Tools needed to run testimage runtime image testing
> -HOSTTOOLS += "${@['', 'ip ping ps scp ssh stty'][bb.data.inherits_class('testimage', d)]}"
> +HOSTTOOLS += "${@['', 'ping ps ssh stty'][bb.data.inherits_class('testimage', d)]}"
>
> # Link to these if present
> HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc sftp socat ssh sudo"
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v5 05/10] bitbake.conf: add ip and scp to HOSTTOOLS unconditionally
2017-10-30 10:51 ` Markus Lehtonen
@ 2017-10-30 13:55 ` Konrad Scherer
2017-10-30 14:42 ` Markus Lehtonen
0 siblings, 1 reply; 41+ messages in thread
From: Konrad Scherer @ 2017-10-30 13:55 UTC (permalink / raw)
To: Markus Lehtonen, ChenQi, openembedded-core
On 10/30/2017 06:51 AM, Markus Lehtonen wrote:
> Hi,
>
> What that tries state is that the image uses facilities written for
> runtimetest (i.e. oeqa.targetcontrol) to run binaries on the target
> machine. The new image introduced (i.e. python-pgo-image) uses this
> to run the profiling task on device.
I ran into something very similar. When using 'TEST_IMAGE = 1' (which
uses testimage-auto), the conditional in bitbake.conf doesn't work and
ip and ping are not available in tmp/hosttools.
When using just 'INHERIT += "testimage"' the conditional works but the
tests are not run automatically.
If I use both 'TEST_IMAGE = 1' and 'INHERIT += "testimage"' together,
the hosttools are found and the tests are run successfully.
This is unexpected because classes/testimage-auto.bbclass does contain
'inherit testimage'.
What config options are you putting in local.conf to trigger the tests?
--
Konrad Scherer, MTS, Linux Products Group, Wind River
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v5 05/10] bitbake.conf: add ip and scp to HOSTTOOLS unconditionally
2017-10-30 13:55 ` Konrad Scherer
@ 2017-10-30 14:42 ` Markus Lehtonen
2017-10-31 6:34 ` ChenQi
0 siblings, 1 reply; 41+ messages in thread
From: Markus Lehtonen @ 2017-10-30 14:42 UTC (permalink / raw)
To: Konrad Scherer, ChenQi, openembedded-core
Hi,
I do not run any runtime tests, I just use the available library functions to run python profiling task on the device. No changes to local.conf are / should be required.
Cheers,
Markus
On 30/10/2017, 15.56, "Konrad Scherer" <Konrad.Scherer@windriver.com> wrote:
On 10/30/2017 06:51 AM, Markus Lehtonen wrote:
> Hi,
>
> What that tries state is that the image uses facilities written for
> runtimetest (i.e. oeqa.targetcontrol) to run binaries on the target
> machine. The new image introduced (i.e. python-pgo-image) uses this
> to run the profiling task on device.
I ran into something very similar. When using 'TEST_IMAGE = 1' (which
uses testimage-auto), the conditional in bitbake.conf doesn't work and
ip and ping are not available in tmp/hosttools.
When using just 'INHERIT += "testimage"' the conditional works but the
tests are not run automatically.
If I use both 'TEST_IMAGE = 1' and 'INHERIT += "testimage"' together,
the hosttools are found and the tests are run successfully.
This is unexpected because classes/testimage-auto.bbclass does contain
'inherit testimage'.
What config options are you putting in local.conf to trigger the tests?
--
Konrad Scherer, MTS, Linux Products Group, Wind River
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v5 05/10] bitbake.conf: add ip and scp to HOSTTOOLS unconditionally
2017-10-30 14:42 ` Markus Lehtonen
@ 2017-10-31 6:34 ` ChenQi
2017-10-31 7:15 ` Markus Lehtonen
0 siblings, 1 reply; 41+ messages in thread
From: ChenQi @ 2017-10-31 6:34 UTC (permalink / raw)
To: Markus Lehtonen, Konrad Scherer, openembedded-core
Konrad,
I've sent out a patch to fix the problem you encountered.
Markus,
I kind of understand what you are doing. You are trying to use library
functions in oeqa. And that requires scp and ip available. Right?
To be clear, I don't have any objection to the any change to HOSTTOOLS.
Just one thing, we need to keep doc in sync.
And one more thing, I cannot find python-pgo-image.
HOSTTOOLS has its default value set in bitbake.conf, and we could modify
it in other conf files like distro conf or local.conf. Could this
possibly be your solution?
Best Regards,
Chen Qi
On 10/30/2017 10:42 PM, Markus Lehtonen wrote:
> Hi,
>
> I do not run any runtime tests, I just use the available library functions to run python profiling task on the device. No changes to local.conf are / should be required.
>
> Cheers,
> Markus
>
> On 30/10/2017, 15.56, "Konrad Scherer" <Konrad.Scherer@windriver.com> wrote:
>
> On 10/30/2017 06:51 AM, Markus Lehtonen wrote:
> > Hi,
> >
> > What that tries state is that the image uses facilities written for
> > runtimetest (i.e. oeqa.targetcontrol) to run binaries on the target
> > machine. The new image introduced (i.e. python-pgo-image) uses this
> > to run the profiling task on device.
>
> I ran into something very similar. When using 'TEST_IMAGE = 1' (which
> uses testimage-auto), the conditional in bitbake.conf doesn't work and
> ip and ping are not available in tmp/hosttools.
>
> When using just 'INHERIT += "testimage"' the conditional works but the
> tests are not run automatically.
>
> If I use both 'TEST_IMAGE = 1' and 'INHERIT += "testimage"' together,
> the hosttools are found and the tests are run successfully.
>
> This is unexpected because classes/testimage-auto.bbclass does contain
> 'inherit testimage'.
>
> What config options are you putting in local.conf to trigger the tests?
>
> --
> Konrad Scherer, MTS, Linux Products Group, Wind River
>
>
>
>
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v5 05/10] bitbake.conf: add ip and scp to HOSTTOOLS unconditionally
2017-10-31 6:34 ` ChenQi
@ 2017-10-31 7:15 ` Markus Lehtonen
2017-11-01 15:35 ` Konrad Scherer
0 siblings, 1 reply; 41+ messages in thread
From: Markus Lehtonen @ 2017-10-31 7:15 UTC (permalink / raw)
To: ChenQi, Konrad Scherer, openembedded-core
Hi,
Please see the 6th patch ("devtools/images: add python-pgo-image") in my patchset for python-pgo-image.
I'd like the profiling to work out-of-the-box, without requiring some strange-ish (i.e. modification of HOSTTOOLS) changes in local.conf. The required utilities, ip and scp, are rather standard tools, anyway. And, they used to be in HOSTTOOLS, unconditionally, before.
Thanks,
Markus
On 31/10/2017, 8.33, "ChenQi" <Qi.Chen@windriver.com> wrote:
Konrad,
I've sent out a patch to fix the problem you encountered.
Markus,
I kind of understand what you are doing. You are trying to use library
functions in oeqa. And that requires scp and ip available. Right?
To be clear, I don't have any objection to the any change to HOSTTOOLS.
Just one thing, we need to keep doc in sync.
And one more thing, I cannot find python-pgo-image.
HOSTTOOLS has its default value set in bitbake.conf, and we could modify
it in other conf files like distro conf or local.conf. Could this
possibly be your solution?
Best Regards,
Chen Qi
On 10/30/2017 10:42 PM, Markus Lehtonen wrote:
> Hi,
>
> I do not run any runtime tests, I just use the available library functions to run python profiling task on the device. No changes to local.conf are / should be required.
>
> Cheers,
> Markus
>
> On 30/10/2017, 15.56, "Konrad Scherer" <Konrad.Scherer@windriver.com> wrote:
>
> On 10/30/2017 06:51 AM, Markus Lehtonen wrote:
> > Hi,
> >
> > What that tries state is that the image uses facilities written for
> > runtimetest (i.e. oeqa.targetcontrol) to run binaries on the target
> > machine. The new image introduced (i.e. python-pgo-image) uses this
> > to run the profiling task on device.
>
> I ran into something very similar. When using 'TEST_IMAGE = 1' (which
> uses testimage-auto), the conditional in bitbake.conf doesn't work and
> ip and ping are not available in tmp/hosttools.
>
> When using just 'INHERIT += "testimage"' the conditional works but the
> tests are not run automatically.
>
> If I use both 'TEST_IMAGE = 1' and 'INHERIT += "testimage"' together,
> the hosttools are found and the tests are run successfully.
>
> This is unexpected because classes/testimage-auto.bbclass does contain
> 'inherit testimage'.
>
> What config options are you putting in local.conf to trigger the tests?
>
> --
> Konrad Scherer, MTS, Linux Products Group, Wind River
>
>
>
>
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v5 05/10] bitbake.conf: add ip and scp to HOSTTOOLS unconditionally
2017-10-31 7:15 ` Markus Lehtonen
@ 2017-11-01 15:35 ` Konrad Scherer
2017-11-15 12:15 ` Markus Lehtonen
2017-11-16 0:13 ` Andre McCurdy
0 siblings, 2 replies; 41+ messages in thread
From: Konrad Scherer @ 2017-11-01 15:35 UTC (permalink / raw)
To: Markus Lehtonen, ChenQi, openembedded-core
On 10/31/2017 03:15 AM, Markus Lehtonen wrote:
> Hi,
>
> Please see the 6th patch ("devtools/images: add python-pgo-image") in my patchset for python-pgo-image.
>
> I'd like the profiling to work out-of-the-box, without requiring some
> strange-ish (i.e. modification of HOSTTOOLS) changes in local.conf.
> The required utilities, ip and scp, are rather standard tools,
> anyway. And, they used to be in HOSTTOOLS, unconditionally, before.
ip and scp are not needed for a normal build so it doesn't make sense to
make them mandatory. Including them unconditionally caused build
failures for us because the docker images we use for coverage builds did
not include those binaries. I realize that doing builds in minimal
containers is rare today, but I expect it to become more common.
Since the recipe uses the oeqa helpers, is there a way to setup the
conditional inclusion to detect this and include the required binaries
in HOSTTOOLS? Or have the oeqa helper modify the HOSTTOOLS before they
are used?
--
Konrad Scherer, MTS, Linux Products Group, Wind River
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v5 05/10] bitbake.conf: add ip and scp to HOSTTOOLS unconditionally
2017-11-01 15:35 ` Konrad Scherer
@ 2017-11-15 12:15 ` Markus Lehtonen
2017-11-16 0:13 ` Andre McCurdy
1 sibling, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-15 12:15 UTC (permalink / raw)
To: Konrad Scherer, ChenQi, openembedded-core
Hi,
Hosttools are set after the bitbake configuration is parsed so there is no way to affect the outcome from the "recipe space". HOSTTOOLS can only be set from bitbake configuration. Thus, I don't see any way to conditionally expand the value in case something specific is being built.
The only way that I can think of to make this somehow work would be to add the utilities to HOSTTOOLS_NONFATAL. Then, the utils would be linked if present and the profile task would succeed on most systems.
Thanks,
Markus
On 01/11/2017, 17.35, "Konrad Scherer" <Konrad.Scherer@windriver.com> wrote:
On 10/31/2017 03:15 AM, Markus Lehtonen wrote:
> Hi,
>
> Please see the 6th patch ("devtools/images: add python-pgo-image") in my patchset for python-pgo-image.
>
> I'd like the profiling to work out-of-the-box, without requiring some
> strange-ish (i.e. modification of HOSTTOOLS) changes in local.conf.
> The required utilities, ip and scp, are rather standard tools,
> anyway. And, they used to be in HOSTTOOLS, unconditionally, before.
ip and scp are not needed for a normal build so it doesn't make sense to
make them mandatory. Including them unconditionally caused build
failures for us because the docker images we use for coverage builds did
not include those binaries. I realize that doing builds in minimal
containers is rare today, but I expect it to become more common.
Since the recipe uses the oeqa helpers, is there a way to setup the
conditional inclusion to detect this and include the required binaries
in HOSTTOOLS? Or have the oeqa helper modify the HOSTTOOLS before they
are used?
--
Konrad Scherer, MTS, Linux Products Group, Wind River
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v5 05/10] bitbake.conf: add ip and scp to HOSTTOOLS unconditionally
2017-11-01 15:35 ` Konrad Scherer
2017-11-15 12:15 ` Markus Lehtonen
@ 2017-11-16 0:13 ` Andre McCurdy
1 sibling, 0 replies; 41+ messages in thread
From: Andre McCurdy @ 2017-11-16 0:13 UTC (permalink / raw)
To: Konrad Scherer; +Cc: OE Core mailing list
On Wed, Nov 1, 2017 at 8:35 AM, Konrad Scherer
<Konrad.Scherer@windriver.com> wrote:
> On 10/31/2017 03:15 AM, Markus Lehtonen wrote:
>> Please see the 6th patch ("devtools/images: add python-pgo-image") in my
>> patchset for python-pgo-image.
>>
>> I'd like the profiling to work out-of-the-box, without requiring some
>> strange-ish (i.e. modification of HOSTTOOLS) changes in local.conf.
>> The required utilities, ip and scp, are rather standard tools,
>> anyway. And, they used to be in HOSTTOOLS, unconditionally, before.
>
> ip and scp are not needed for a normal build so it doesn't make sense to
> make them mandatory.
Shouldn't the preferred approach be to use -native tools whenever
possible? It solves the problem of needing to add obscure tools into
HOSTTOOLS and also makes tests more reproducible by ensuring that they
always run with a known version of a particular tool, not something
which may be host specific.
> Including them unconditionally caused build failures
> for us because the docker images we use for coverage builds did not include
> those binaries. I realize that doing builds in minimal containers is rare
> today, but I expect it to become more common.
It's not so uncommon, even now.
> Since the recipe uses the oeqa helpers, is there a way to setup the
> conditional inclusion to detect this and include the required binaries in
> HOSTTOOLS? Or have the oeqa helper modify the HOSTTOOLS before they are
> used?
>
>
> --
> Konrad Scherer, MTS, Linux Products Group, Wind River
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v6 00/15] support profile-optimized build for Python
2017-10-27 10:07 [PATCH v5 00/10] support profile-optimized build for Python Markus Lehtonen
` (10 preceding siblings ...)
2017-10-27 10:34 ` ✗ patchtest: failure for support profile-optimized build for Python Patchwork
@ 2017-11-20 12:15 ` Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 01/15] python3-native: support profile optimized build Markus Lehtonen
` (14 more replies)
2017-11-20 12:33 ` ✗ patchtest: failure for support profile-optimized build for Python (rev2) Patchwork
12 siblings, 15 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-20 12:15 UTC (permalink / raw)
To: openembedded-core
Sixth attempt on enabling profile-guided-optimization for Python 3.
Changes since v5:
- drop the patch adding 'scp' and 'ip' to HOSTTOOLS
- but, enable openssh-native and iproute2-native (and their native build deps)
and use these, instead
- dependencies of 'do_profile' task of python-pgo-image are managed via DEPENDS
instead of task depends (which were used in a bit awkward way in the previous
version of the patchset)
- re-introduce get_target_controller() function into oeqa/targetcontrol
- one additional (unrelated) bugfix to oeqa/targetcontrol
The following changes since commit a17f3ec910366e9e7551fa24fbc07929b9584341:
dhcp: fix build issue with libxml2 support (2017-11-10 14:44:31 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib marquiz/fixes-9338
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=marquiz/fixes-9338
Markus Lehtonen (15):
python3-native: support profile optimized build
python3: fix depends of python3-tests
python3: add python3-profile-opt recipe
python3-profile-opt: rename libpython3
openssh: extend to -native
libmnl: enable native
iptables: enable native
iproute2: enable native
oeqa/targetcontrol: re-introduce get_target_controller()
oeqa/targetcontrol: add missing arg to SimpleRemoteTarget.__init__
devtools/images: add python-pgo-image
python3: support profile optimized build
python3: fix profile-optimized build of modules
python3: add python3-tools subpackage
python3: remove two setup.py cross-compile hacks
meta/lib/oeqa/targetcontrol.py | 29 +++++-
meta/recipes-connectivity/iproute2/iproute2.inc | 7 +-
...1-don-t-use-absolute-path-for-ssh-program.patch | 31 ++++++
meta/recipes-connectivity/openssh/openssh_7.6p1.bb | 7 +-
meta/recipes-devtools/images/python-pgo-image.bb | 92 ++++++++++++++++++
.../python/python-3.5-manifest.inc | 10 +-
.../python/python3-native_3.5.3.bb | 9 ++
meta/recipes-devtools/python/python3-profile-opt | 1 +
.../python/python3-profile-opt_3.5.3.bb | 10 ++
.../python3/0001-cross-compile-support.patch | 9 --
...efile-add-install_generate_profile-target.patch | 25 +++++
...-CFLAGS-for-extensions-when-cross-compili.patch | 56 +++++++++++
.../python3-setup.py-no-host-headers-libs.patch | 33 -------
...name-libpython3-to-libpython-profile-opt3.patch | 108 +++++++++++++++++++++
.../setup.py-find-libraries-in-staging-dirs.patch | 30 ------
meta/recipes-devtools/python/python3_3.5.3.bb | 58 ++++++++---
...revent-absolute-path-in-installed-symlink.patch | 29 ++++++
meta/recipes-extended/iptables/iptables_1.6.1.bb | 3 +
meta/recipes-extended/libmnl/libmnl_1.0.4.bb | 2 +
19 files changed, 453 insertions(+), 96 deletions(-)
create mode 100644 meta/recipes-connectivity/openssh/openssh/0001-don-t-use-absolute-path-for-ssh-program.patch
create mode 100644 meta/recipes-devtools/images/python-pgo-image.bb
create mode 120000 meta/recipes-devtools/python/python3-profile-opt
create mode 100644 meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
create mode 100644 meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch
create mode 100644 meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch
delete mode 100644 meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch
create mode 100644 meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch
delete mode 100644 meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
create mode 100644 meta/recipes-extended/iptables/iptables/prevent-absolute-path-in-installed-symlink.patch
--
2.13.6
^ permalink raw reply [flat|nested] 41+ messages in thread
* [PATCH v6 01/15] python3-native: support profile optimized build
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
@ 2017-11-20 12:15 ` Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 02/15] python3: fix depends of python3-tests Markus Lehtonen
` (13 subsequent siblings)
14 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-20 12:15 UTC (permalink / raw)
To: openembedded-core
Make it possible to build python3-native with profile directed optimization
enabled. The feature is enabled by specifying
PYTHON3_NATIVE_PROFILE_OPT = "1" in local.conf.
The profile task to be run may be specified with PYTHON3_NATIVE_PROFILE_TASK
variable in local.conf, e.g.
PYTHON3_NATIVE_PROFILE_TASK = "${S}/Tools/pybench/pybench.py -n 2 --with-gc --with-syscheck"
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python3-native_3.5.3.bb | 9 +++++++++
.../python/python3/0001-cross-compile-support.patch | 9 ---------
2 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/meta/recipes-devtools/python/python3-native_3.5.3.bb b/meta/recipes-devtools/python/python3-native_3.5.3.bb
index 8cd9c88a82..936ea05b29 100644
--- a/meta/recipes-devtools/python/python3-native_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3-native_3.5.3.bb
@@ -58,11 +58,20 @@ EXTRA_OEMAKE = '\
# No ctypes option for python 3
PYTHONLSBOPTS = ""
+PYTHON3_NATIVE_MAKE_TARGET ?= "${@'profile-opt' if d.getVar('PYTHON3_NATIVE_PROFILE_OPT', True) == '1' else ''}"
+
do_configure_append() {
autoreconf --verbose --install --force --exclude=autopoint ../Python-${PV}/Modules/_ctypes/libffi
sed -i -e 's,#define HAVE_GETRANDOM 1,/\* #undef HAVE_GETRANDOM \*/,' ${B}/pyconfig.h
}
+do_compile() {
+ if [ -n "${PYTHON3_NATIVE_PROFILE_TASK}" ]; then
+ sed -i -e 's,^PROFILE_TASK=.*,PROFILE_TASK=${PYTHON3_NATIVE_PROFILE_TASK},g' Makefile
+ fi
+ oe_runmake ${PYTHON3_NATIVE_MAKE_TARGET}
+}
+
do_install() {
install -d ${D}${libdir}/pkgconfig
oe_runmake 'DESTDIR=${D}' install
diff --git a/meta/recipes-devtools/python/python3/0001-cross-compile-support.patch b/meta/recipes-devtools/python/python3/0001-cross-compile-support.patch
index 118d75ddc5..bd599a8b7e 100644
--- a/meta/recipes-devtools/python/python3/0001-cross-compile-support.patch
+++ b/meta/recipes-devtools/python/python3/0001-cross-compile-support.patch
@@ -34,15 +34,6 @@ index a88b7d5..7cb8bb3 100644
PSRCS= \
Parser/acceler.c \
-@@ -510,7 +512,7 @@ build_all_generate_profile:
-
- run_profile_task:
- : # FIXME: can't run for a cross build
-- $(LLVM_PROF_FILE) $(RUNSHARED) ./$(BUILDPYTHON) $(PROFILE_TASK) || true
-+ $(LLVM_PROF_FILE) $(RUNSHARED) $(HOSTPYTHON) $(PROFILE_TASK) || true
-
- build_all_merge_profile:
- $(LLVM_PROF_MERGER)
@@ -787,7 +789,7 @@ $(IO_OBJS): $(IO_H)
$(GRAMMAR_H): @GENERATED_COMMENT@ $(GRAMMAR_INPUT) $(PGEN)
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v6 02/15] python3: fix depends of python3-tests
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 01/15] python3-native: support profile optimized build Markus Lehtonen
@ 2017-11-20 12:15 ` Markus Lehtonen
2017-11-20 23:37 ` Tim Orling
2017-11-20 12:15 ` [PATCH v6 03/15] python3: add python3-profile-opt recipe Markus Lehtonen
` (12 subsequent siblings)
14 siblings, 1 reply; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-20 12:15 UTC (permalink / raw)
To: openembedded-core
Similar to an earlier fix for Python 2.7. Make the tests subpackage
depend on all modules as test.regrtest uses most (if not all) of them.
Some tests also depend on libgcc so add that as a runtime dependency as
well.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python-3.5-manifest.inc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/python/python-3.5-manifest.inc b/meta/recipes-devtools/python/python-3.5-manifest.inc
index 0260e87e75..067f6c58c5 100644
--- a/meta/recipes-devtools/python/python-3.5-manifest.inc
+++ b/meta/recipes-devtools/python/python-3.5-manifest.inc
@@ -241,7 +241,7 @@ RDEPENDS_${PN}-terminal="${PN}-core ${PN}-io"
FILES_${PN}-terminal="${libdir}/python3.5/pty.* ${libdir}/python3.5/__pycache__/pty.* ${libdir}/python3.5/tty.* ${libdir}/python3.5/__pycache__/tty.* "
SUMMARY_${PN}-tests="Python tests"
-RDEPENDS_${PN}-tests="${PN}-core ${PN}-compression"
+RDEPENDS_${PN}-tests="${PN}-core ${PN}-modules libgcc"
FILES_${PN}-tests="${libdir}/python3.5/test ${libdir}/python3.5/test/__pycache__ "
SUMMARY_${PN}-textutils="Python option parsing, text wrapping and CSV support"
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v6 03/15] python3: add python3-profile-opt recipe
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 01/15] python3-native: support profile optimized build Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 02/15] python3: fix depends of python3-tests Markus Lehtonen
@ 2017-11-20 12:15 ` Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 04/15] python3-profile-opt: rename libpython3 Markus Lehtonen
` (11 subsequent siblings)
14 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-20 12:15 UTC (permalink / raw)
To: openembedded-core
This patch adds a new recipe that builds a special version of python3 that
produces profile data used for optimization. The new recipe directly includes
the base python recipe so that they are build in as similar way as possible and
this hopefully decreases the recipe maintenance burden, too. Also, its files
clash with the "normal" python3 so they cannot be installed in an image at the
same time.
Normally, profile-guided-optimization in Python is done simply by doing "make
profile-opt" which first builds python with profile instrumentation enabled,
then runs a profile task to get the profile data, and last, re-builds python
with profile data guiding the optimization. However, in our cross-build
environment this gets a lot trickier. We need to split out the steps as
building is done on the build host but we need to run the second step (i.e. run
the profile task) on the target hardware. This patch enables the first step,
i.e. building python with profile instrumentation enabled
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python3-profile-opt | 1 +
.../python/python3-profile-opt_3.5.3.bb | 9 +++++++
...efile-add-install_generate_profile-target.patch | 25 ++++++++++++++++++
meta/recipes-devtools/python/python3_3.5.3.bb | 30 ++++++++++++----------
4 files changed, 51 insertions(+), 14 deletions(-)
create mode 120000 meta/recipes-devtools/python/python3-profile-opt
create mode 100644 meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
create mode 100644 meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch
diff --git a/meta/recipes-devtools/python/python3-profile-opt b/meta/recipes-devtools/python/python3-profile-opt
new file mode 120000
index 0000000000..b8a0adbbb9
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-profile-opt
@@ -0,0 +1 @@
+python3
\ No newline at end of file
diff --git a/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb b/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
new file mode 100644
index 0000000000..6940480546
--- /dev/null
+++ b/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
@@ -0,0 +1,9 @@
+require python3_${PV}.bb
+
+SRC_URI += "file://Makefile-add-install_generate_profile-target.patch \
+ "
+
+PYTHON3_MAKE_TARGET = "build_all_generate_profile"
+
+RCONFLICTS_${PN}-core = "python3-core"
+RCONFLICTS_lib${BPN} = "libpython3"
diff --git a/meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch b/meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch
new file mode 100644
index 0000000000..7aee7fb799
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/Makefile-add-install_generate_profile-target.patch
@@ -0,0 +1,25 @@
+Makefile: add install_generate_profile target
+
+Making it possible to install the binaries with profile generation
+enabled. We need this as linking is also performed at the install stage.
+
+Upstream-Status: Inappropriate [oe-specific]
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 2b87614..826155b 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1046,6 +1046,8 @@ QUICKTESTOPTS= $(TESTOPTS) -x test_subprocess test_io test_lib2to3 \
+ quicktest: all platform
+ $(TESTRUNNER) $(QUICKTESTOPTS)
+
++install_generate_profile:
++ $(MAKE) install LDFLAGS="$(LDFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LIBS="$(LIBS)"
+
+ install: @FRAMEWORKINSTALLFIRST@ commoninstall bininstall maninstall @FRAMEWORKINSTALLLAST@
+ if test "x$(ENSUREPIP)" != "xno" ; then \
+--
+2.6.6
+
diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
index 281a65f784..bb524eb17d 100644
--- a/meta/recipes-devtools/python/python3_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3_3.5.3.bb
@@ -115,14 +115,10 @@ do_compile() {
# then call do_install twice we get Makefile.orig == Makefile.sysroot
install -m 0644 Makefile Makefile.sysroot
- oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
- HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python3-native/python3 \
- STAGING_LIBDIR=${STAGING_LIBDIR} \
- STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
- STAGING_INCDIR=${STAGING_INCDIR} \
- LIB=${baselib} \
- ARCH=${TARGET_ARCH} \
- OPT="${CFLAGS}" libpython3.so
+ if [ "${PYTHON3_MAKE_TARGET}" = "build_all_generate_profile" ]; then
+ # This is only used in PGO profiling by python3-profile-opt package
+ export EXTRA_CFLAGS="-fprofile-dir=./python3-pgo-profiles/"
+ fi
oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python3-native/python3 \
@@ -131,7 +127,7 @@ do_compile() {
STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
LIB=${baselib} \
ARCH=${TARGET_ARCH} \
- OPT="${CFLAGS}"
+ OPT="${CFLAGS}" ${PYTHON3_MAKE_TARGET}
}
do_install() {
@@ -151,8 +147,14 @@ do_install() {
STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
LIB=${baselib} \
ARCH=${TARGET_ARCH} \
- DESTDIR=${D} LIBDIR=${libdir}
+ DESTDIR=${D} LIBDIR=${libdir} ${PYTHON3_MAKE_TARGET}
+ if [ "${PYTHON3_MAKE_TARGET}" = "build_all_generate_profile" ]; then
+ # Need special make install if pgo generation is enabled
+ _PYTHON3_MAKE_INSTALL_TARGET="install_generate_profile"
+ else
+ _PYTHON3_MAKE_INSTALL_TARGET="install"
+ fi
oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
HOSTPYTHON=${STAGING_BINDIR_NATIVE}/python3-native/python3 \
STAGING_LIBDIR=${STAGING_LIBDIR} \
@@ -160,7 +162,7 @@ do_install() {
STAGING_BASELIBDIR=${STAGING_BASELIBDIR} \
LIB=${baselib} \
ARCH=${TARGET_ARCH} \
- DESTDIR=${D} LIBDIR=${libdir} install
+ DESTDIR=${D} LIBDIR=${libdir} ${_PYTHON3_MAKE_INSTALL_TARGET}
# avoid conflict with 2to3 from Python 2
rm -f ${D}/${bindir}/2to3
@@ -209,9 +211,9 @@ PACKAGES =+ "${PN}-pyvenv"
FILES_${PN}-pyvenv += "${bindir}/pyvenv-${PYTHON_MAJMIN} ${bindir}/pyvenv"
# package libpython3
-PACKAGES =+ "libpython3 libpython3-staticdev"
-FILES_libpython3 = "${libdir}/libpython*.so.*"
-FILES_libpython3-staticdev += "${libdir}/python${PYTHON_MAJMIN}/config-${PYTHON_BINABI}/libpython${PYTHON_BINABI}.a"
+PACKAGES =+ "lib${BPN} lib${BPN}-staticdev"
+FILES_lib${BPN} = "${libdir}/libpython*.so.*"
+FILES_lib${BPN}-staticdev += "${libdir}/python${PYTHON_MAJMIN}/config-${PYTHON_BINABI}/libpython*.a"
INSANE_SKIP_${PN}-dev += "dev-elf"
# catch all the rest (unsorted)
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v6 04/15] python3-profile-opt: rename libpython3
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
` (2 preceding siblings ...)
2017-11-20 12:15 ` [PATCH v6 03/15] python3: add python3-profile-opt recipe Markus Lehtonen
@ 2017-11-20 12:15 ` Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 05/15] openssh: extend to -native Markus Lehtonen
` (10 subsequent siblings)
14 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-20 12:15 UTC (permalink / raw)
To: openembedded-core
Prevents soname clash with "normal" libpython3. The python library
needs to be renamed so that the automatic ndency generation/checking
mechanism in bitbake does not get confused. Otherwise
python3-profile-opt will depend on libpython3 from the "normal" python3
package.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
.../python/python3-profile-opt_3.5.3.bb | 1 +
...name-libpython3-to-libpython-profile-opt3.patch | 108 +++++++++++++++++++++
2 files changed, 109 insertions(+)
create mode 100644 meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch
diff --git a/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb b/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
index 6940480546..3ad427117f 100644
--- a/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3-profile-opt_3.5.3.bb
@@ -1,6 +1,7 @@
require python3_${PV}.bb
SRC_URI += "file://Makefile-add-install_generate_profile-target.patch \
+ file://rename-libpython3-to-libpython-profile-opt3.patch \
"
PYTHON3_MAKE_TARGET = "build_all_generate_profile"
diff --git a/meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch b/meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch
new file mode 100644
index 0000000000..2e5f9302fc
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/rename-libpython3-to-libpython-profile-opt3.patch
@@ -0,0 +1,108 @@
+rename libpython3 to libpython-profile-opt3
+
+Upstream-Status: Inappropriate [oe-specific]
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+
+diff --git a/Lib/distutils/command/build_ext.py b/Lib/distutils/command/build_ext.py
+index 7c278ef..78613d8 100644
+--- a/Lib/distutils/command/build_ext.py
++++ b/Lib/distutils/command/build_ext.py
+@@ -747,7 +747,7 @@ class build_ext(Command):
+ else:
+ from distutils import sysconfig
+ if sysconfig.get_config_var('Py_ENABLE_SHARED'):
+- pythonlib = 'python{}.{}{}'.format(
++ pythonlib = 'python-profile-opt{}.{}{}'.format(
+ sys.hexversion >> 24, (sys.hexversion >> 16) & 0xff,
+ sysconfig.get_config_var('ABIFLAGS'))
+ return ext.libraries + [pythonlib]
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 0327385..f744ce1 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -623,7 +623,7 @@ $(LIBRARY): $(LIBRARY_OBJS)
+ $(AR) $(ARFLAGS) $@ $(MODOBJS)
+ $(RANLIB) $@
+
+-libpython$(LDVERSION).so: $(LIBRARY_OBJS)
++libpython-profile-opt$(LDVERSION).so: $(LIBRARY_OBJS)
+ if test $(INSTSONAME) != $(LDLIBRARY); then \
+ $(BLDSHARED) -Wl,-h$(INSTSONAME) -o $(INSTSONAME) $(LIBRARY_OBJS) $(MODLIBS) $(SHLIBS) $(LIBC) $(LIBM) $(LDLAST); \
+ $(LN) -f $(INSTSONAME) $@; \
+@@ -631,7 +631,7 @@ libpython$(LDVERSION).so: $(LIBRARY_OBJS)
+ $(BLDSHARED) -o $@ $(LIBRARY_OBJS) $(MODLIBS) $(SHLIBS) $(LIBC) $(LIBM) $(LDLAST); \
+ fi
+
+-libpython3.so: libpython$(LDVERSION).so
++libpython-profile-opt3.so: libpython-profile-opt$(LDVERSION).so
+ $(BLDSHARED) $(NO_AS_NEEDED) -o $@ -Wl,-h$@ $^
+
+ libpython$(LDVERSION).dylib: $(LIBRARY_OBJS)
+diff --git a/Misc/python-config.in b/Misc/python-config.in
+index a808f1a..0dcc615 100644
+--- a/Misc/python-config.in
++++ b/Misc/python-config.in
+@@ -47,7 +47,7 @@ for opt in opt_flags:
+ print(' '.join(flags))
+
+ elif opt in ('--libs', '--ldflags'):
+- libs = ['-lpython' + pyver + sys.abiflags]
++ libs = ['-lpython-profile-opt' + pyver + sys.abiflags]
+ libs += getvar('LIBS').split()
+ libs += getvar('SYSLIBS').split()
+ # add the prefix/lib/pythonX.Y/config dir, but only if there is no
+diff --git a/Misc/python-config.sh.in b/Misc/python-config.sh.in
+index 30c6927..efad748 100644
+--- a/Misc/python-config.sh.in
++++ b/Misc/python-config.sh.in
+@@ -40,7 +40,7 @@ LIBM="@LIBM@"
+ LIBC="@LIBC@"
+ SYSLIBS="$LIBM $LIBC"
+ ABIFLAGS="@ABIFLAGS@"
+-LIBS="-lpython${VERSION}${ABIFLAGS} @LIBS@ $SYSLIBS"
++LIBS="-lpython-profile-opt${VERSION}${ABIFLAGS} @LIBS@ $SYSLIBS"
+ BASECFLAGS="@BASECFLAGS@"
+ LDLIBRARY="@LDLIBRARY@"
+ LINKFORSHARED="@LINKFORSHARED@"
+diff --git a/Misc/python.pc.in b/Misc/python.pc.in
+index ae69867..b73d027 100644
+--- a/Misc/python.pc.in
++++ b/Misc/python.pc.in
+@@ -9,5 +9,5 @@ Description: Python library
+ Requires:
+ Version: @VERSION@
+ Libs.private: @LIBS@
+-Libs: -L${libdir} -lpython@VERSION@@ABIFLAGS@
++Libs: -L${libdir} -lpython-profile-opt@VERSION@@ABIFLAGS@
+ Cflags: -I${includedir}/python@VERSION@@ABIFLAGS@
+diff --git a/configure.ac b/configure.ac
+index d8c2847..ad63e4e 100644
+--- a/configure.ac
++++ b/configure.ac
+@@ -1001,7 +1001,7 @@ AC_SUBST(LIBRARY)
+ AC_MSG_CHECKING(LIBRARY)
+ if test -z "$LIBRARY"
+ then
+- LIBRARY='libpython$(VERSION)$(ABIFLAGS).a'
++ LIBRARY='libpython-profile-opt$(VERSION)$(ABIFLAGS).a'
+ fi
+ AC_MSG_RESULT($LIBRARY)
+
+@@ -1162,13 +1162,13 @@ if test $enable_shared = "yes"; then
+ fi
+ ;;
+ Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*|OpenBSD*)
+- LDLIBRARY='libpython$(LDVERSION).so'
+- BLDLIBRARY='-L. -lpython$(LDVERSION)'
++ LDLIBRARY='libpython-profile-opt$(LDVERSION).so'
++ BLDLIBRARY='-L. -lpython-profile-opt$(LDVERSION)'
+ RUNSHARED=LD_LIBRARY_PATH=`pwd`${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
+ INSTSONAME="$LDLIBRARY".$SOVERSION
+ if test "$with_pydebug" != yes
+ then
+- PY3LIBRARY=libpython3.so
++ PY3LIBRARY=libpython-profile-opt3.so
+ fi
+ ;;
+ hp*|HP*)
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v6 05/15] openssh: extend to -native
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
` (3 preceding siblings ...)
2017-11-20 12:15 ` [PATCH v6 04/15] python3-profile-opt: rename libpython3 Markus Lehtonen
@ 2017-11-20 12:15 ` Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 06/15] libmnl: enable native Markus Lehtonen
` (9 subsequent siblings)
14 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-20 12:15 UTC (permalink / raw)
To: openembedded-core
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
...1-don-t-use-absolute-path-for-ssh-program.patch | 31 ++++++++++++++++++++++
meta/recipes-connectivity/openssh/openssh_7.6p1.bb | 7 ++++-
2 files changed, 37 insertions(+), 1 deletion(-)
create mode 100644 meta/recipes-connectivity/openssh/openssh/0001-don-t-use-absolute-path-for-ssh-program.patch
diff --git a/meta/recipes-connectivity/openssh/openssh/0001-don-t-use-absolute-path-for-ssh-program.patch b/meta/recipes-connectivity/openssh/openssh/0001-don-t-use-absolute-path-for-ssh-program.patch
new file mode 100644
index 0000000000..7c15dd72c8
--- /dev/null
+++ b/meta/recipes-connectivity/openssh/openssh/0001-don-t-use-absolute-path-for-ssh-program.patch
@@ -0,0 +1,31 @@
+From fb78bd8142f48d0909d8839e8bab48abe39d5ab7 Mon Sep 17 00:00:00 2001
+From: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+Date: Mon, 20 Nov 2017 12:08:56 +0200
+Subject: [PATCH] don't use absolute path for ssh program
+
+Makes it possible to relocate openssh binaries referencing to the ssh
+binary.
+
+Utream-Status: Inappropriate [oe-specific]
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+---
+ Makefile.in | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/Makefile.in b/Makefile.in
+index c52ce19..752f7d0 100644
+--- a/Makefile.in
++++ b/Makefile.in
+@@ -21,7 +21,7 @@ top_srcdir=@top_srcdir@
+
+ DESTDIR=
+ VPATH=@srcdir@
+-SSH_PROGRAM=@bindir@/ssh
++SSH_PROGRAM=ssh
+ ASKPASS_PROGRAM=$(libexecdir)/ssh-askpass
+ SFTP_SERVER=$(libexecdir)/sftp-server
+ SSH_KEYSIGN=$(libexecdir)/ssh-keysign
+--
+2.13.6
+
diff --git a/meta/recipes-connectivity/openssh/openssh_7.6p1.bb b/meta/recipes-connectivity/openssh/openssh_7.6p1.bb
index ebb9a5734d..b0b9ba3f01 100644
--- a/meta/recipes-connectivity/openssh/openssh_7.6p1.bb
+++ b/meta/recipes-connectivity/openssh/openssh_7.6p1.bb
@@ -27,6 +27,8 @@ SRC_URI = "http://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-${PV}.tar
file://add-test-support-for-busybox.patch \
"
+SRC_URI_append_class-native = " file://0001-don-t-use-absolute-path-for-ssh-program.patch"
+
PAM_SRC_URI = "file://sshd"
SRC_URI[md5sum] = "06a88699018e5fef13d4655abfed1f63"
@@ -52,7 +54,7 @@ CFLAGS += "-D__FILE_OFFSET_BITS=64"
EXTRA_OECONF = "'LOGIN_PROGRAM=${base_bindir}/login' \
${@bb.utils.contains('DISTRO_FEATURES', 'pam', '--with-pam', '--without-pam', d)} \
--without-zlib-version-check \
- --with-privsep-path=/var/run/sshd \
+ --with-privsep-path=${localstatedir}/run/sshd \
--sysconfdir=${sysconfdir}/ssh \
--with-xauth=/usr/bin/xauth \
--disable-strip \
@@ -149,6 +151,8 @@ RDEPENDS_${PN} += "${PN}-scp ${PN}-ssh ${PN}-sshd ${PN}-keygen"
RDEPENDS_${PN}-sshd += "${PN}-keygen ${@bb.utils.contains('DISTRO_FEATURES', 'pam', 'pam-plugin-keyinit pam-plugin-loginuid', '', d)}"
RDEPENDS_${PN}-ptest += "${PN}-sftp ${PN}-misc ${PN}-sftp-server make"
+RDEPENDS_${PN}_class-native = ""
+
RPROVIDES_${PN}-ssh = "ssh"
RPROVIDES_${PN}-sshd = "sshd"
@@ -163,3 +167,4 @@ ALTERNATIVE_PRIORITY = "90"
ALTERNATIVE_${PN}-scp = "scp"
ALTERNATIVE_${PN}-ssh = "ssh"
+BBCLASSEXTEND = "native"
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v6 06/15] libmnl: enable native
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
` (4 preceding siblings ...)
2017-11-20 12:15 ` [PATCH v6 05/15] openssh: extend to -native Markus Lehtonen
@ 2017-11-20 12:15 ` Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 07/15] iptables: " Markus Lehtonen
` (8 subsequent siblings)
14 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-20 12:15 UTC (permalink / raw)
To: openembedded-core
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-extended/libmnl/libmnl_1.0.4.bb | 2 ++
1 file changed, 2 insertions(+)
diff --git a/meta/recipes-extended/libmnl/libmnl_1.0.4.bb b/meta/recipes-extended/libmnl/libmnl_1.0.4.bb
index b458799800..23262e8c6f 100644
--- a/meta/recipes-extended/libmnl/libmnl_1.0.4.bb
+++ b/meta/recipes-extended/libmnl/libmnl_1.0.4.bb
@@ -11,3 +11,5 @@ SRC_URI[tar.md5sum] = "be9b4b5328c6da1bda565ac5dffadb2d"
SRC_URI[tar.sha256sum] = "171f89699f286a5854b72b91d06e8f8e3683064c5901fb09d954a9ab6f551f81"
inherit autotools pkgconfig
+
+BBCLASSEXTEND = "native"
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v6 07/15] iptables: enable native
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
` (5 preceding siblings ...)
2017-11-20 12:15 ` [PATCH v6 06/15] libmnl: enable native Markus Lehtonen
@ 2017-11-20 12:15 ` Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 08/15] iproute2: " Markus Lehtonen
` (7 subsequent siblings)
14 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-20 12:15 UTC (permalink / raw)
To: openembedded-core
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
...revent-absolute-path-in-installed-symlink.patch | 29 ++++++++++++++++++++++
meta/recipes-extended/iptables/iptables_1.6.1.bb | 3 +++
2 files changed, 32 insertions(+)
create mode 100644 meta/recipes-extended/iptables/iptables/prevent-absolute-path-in-installed-symlink.patch
diff --git a/meta/recipes-extended/iptables/iptables/prevent-absolute-path-in-installed-symlink.patch b/meta/recipes-extended/iptables/iptables/prevent-absolute-path-in-installed-symlink.patch
new file mode 100644
index 0000000000..3ca6c0dd28
--- /dev/null
+++ b/meta/recipes-extended/iptables/iptables/prevent-absolute-path-in-installed-symlink.patch
@@ -0,0 +1,29 @@
+From d7a50cf5a9f03f3b65293d556dadfc484456b0d8 Mon Sep 17 00:00:00 2001
+From: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+Date: Fri, 17 Nov 2017 14:10:59 +0200
+Subject: [PATCH] prevent absolute path in installed symlink
+
+Upstream-Status: Inappropriate [oe-specific]
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+---
+ iptables/Makefile.am | 3 ++-
+ 1 file changed, 2 insertions(+), 1 deletion(-)
+
+diff --git a/iptables/Makefile.am b/iptables/Makefile.am
+index f92cc4f..55ce793 100644
+--- a/iptables/Makefile.am
++++ b/iptables/Makefile.am
+@@ -88,7 +88,8 @@ pkgconfig_DATA = xtables.pc
+ install-exec-hook:
+ -if test -z "${DESTDIR}"; then /sbin/ldconfig; fi;
+ ${INSTALL} -dm0755 "${DESTDIR}${bindir}";
+- for i in ${vx_bin_links}; do ${LN_S} -f "${sbindir}/xtables-multi" "${DESTDIR}${bindir}/$$i"; done;
++ _relpath=`python -c "import os.path; print(os.path.relpath('${sbindir}/xtables-multi', '${bindir}'))"`; \
++ for i in ${vx_bin_links}; do ${LN_S} -f "$${_relpath}" "${DESTDIR}${bindir}/$$i"; done;
+ for i in ${v4_sbin_links}; do ${LN_S} -f xtables-multi "${DESTDIR}${sbindir}/$$i"; done;
+ for i in ${v6_sbin_links}; do ${LN_S} -f xtables-multi "${DESTDIR}${sbindir}/$$i"; done;
+ for i in ${x_sbin_links}; do ${LN_S} -f xtables-compat-multi "${DESTDIR}${sbindir}/$$i"; done;
+--
+2.13.6
+
diff --git a/meta/recipes-extended/iptables/iptables_1.6.1.bb b/meta/recipes-extended/iptables/iptables_1.6.1.bb
index b37c55a64e..2f565821a4 100644
--- a/meta/recipes-extended/iptables/iptables_1.6.1.bb
+++ b/meta/recipes-extended/iptables/iptables_1.6.1.bb
@@ -23,6 +23,7 @@ SRC_URI = "http://netfilter.org/projects/iptables/files/iptables-${PV}.tar.bz2 \
file://0001-configure-Add-option-to-enable-disable-libnfnetlink.patch \
file://0002-configure.ac-only-check-conntrack-when-libnfnetlink-enabled.patch \
file://0001-fix-build-with-musl.patch \
+ file://prevent-absolute-path-in-installed-symlink.patch \
"
SRC_URI[md5sum] = "ab38a33806b6182c6f53d6afb4619add"
@@ -47,3 +48,5 @@ do_configure_prepend() {
# Keep ax_check_linker_flags.m4 which belongs to autoconf-archive.
rm -f libtool.m4 lt~obsolete.m4 ltoptions.m4 ltsugar.m4 ltversion.m4
}
+
+BBCLASSEXTEND = "native"
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v6 08/15] iproute2: enable native
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
` (6 preceding siblings ...)
2017-11-20 12:15 ` [PATCH v6 07/15] iptables: " Markus Lehtonen
@ 2017-11-20 12:15 ` Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 09/15] oeqa/targetcontrol: re-introduce get_target_controller() Markus Lehtonen
` (6 subsequent siblings)
14 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-20 12:15 UTC (permalink / raw)
To: openembedded-core
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-connectivity/iproute2/iproute2.inc | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/meta/recipes-connectivity/iproute2/iproute2.inc b/meta/recipes-connectivity/iproute2/iproute2.inc
index a578eb3afa..d08d049f41 100644
--- a/meta/recipes-connectivity/iproute2/iproute2.inc
+++ b/meta/recipes-connectivity/iproute2/iproute2.inc
@@ -26,12 +26,15 @@ do_configure_append () {
do_install () {
oe_runmake DESTDIR=${D} install
- mv ${D}${base_sbindir}/ip ${D}${base_sbindir}/ip.iproute2
install -d ${D}${datadir}
mv ${D}/share/* ${D}${datadir}/ || true
rm ${D}/share -rf || true
}
+do_install_append_class-target () {
+ mv ${D}${base_sbindir}/ip ${D}${base_sbindir}/ip.iproute2
+}
+
# The .so files in iproute2-tc are modules, not traditional libraries
INSANE_SKIP_${PN}-tc = "dev-so"
@@ -50,3 +53,5 @@ ALTERNATIVE_${PN} = "ip"
ALTERNATIVE_TARGET[ip] = "${base_sbindir}/ip.${BPN}"
ALTERNATIVE_LINK_NAME[ip] = "${base_sbindir}/ip"
ALTERNATIVE_PRIORITY = "100"
+
+BBCLASSEXTEND = "native"
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v6 09/15] oeqa/targetcontrol: re-introduce get_target_controller()
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
` (7 preceding siblings ...)
2017-11-20 12:15 ` [PATCH v6 08/15] iproute2: " Markus Lehtonen
@ 2017-11-20 12:15 ` Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 10/15] oeqa/targetcontrol: add missing arg to SimpleRemoteTarget.__init__ Markus Lehtonen
` (5 subsequent siblings)
14 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-20 12:15 UTC (permalink / raw)
To: openembedded-core
This function was dropped in f352ef37a27cb0871c87cf9991d8ad2e1d560fde as
unused, but, python-pgo-image needs it for running profile task on
target hw.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/lib/oeqa/targetcontrol.py | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/meta/lib/oeqa/targetcontrol.py b/meta/lib/oeqa/targetcontrol.py
index f63936c3ec..d147a44480 100644
--- a/meta/lib/oeqa/targetcontrol.py
+++ b/meta/lib/oeqa/targetcontrol.py
@@ -18,6 +18,31 @@ from oeqa.utils.dump import TargetDumper
from oeqa.controllers.testtargetloader import TestTargetLoader
from abc import ABCMeta, abstractmethod
+def get_target_controller(d, logger):
+ testtarget = d.getVar("TEST_TARGET")
+ # old, simple names
+ if testtarget == "qemu":
+ return QemuTarget(d, logger)
+ elif testtarget == "simpleremote":
+ return SimpleRemoteTarget(d, logger)
+ else:
+ # use the class name
+ try:
+ # is it a core class defined here?
+ controller = getattr(sys.modules[__name__], testtarget)
+ except AttributeError:
+ # nope, perhaps a layer defined one
+ try:
+ bbpath = d.getVar("BBPATH").split(':')
+ testtargetloader = TestTargetLoader()
+ controller = testtargetloader.get_controller_module(testtarget, bbpath)
+ except ImportError as e:
+ bb.fatal("Failed to import {0} from available controller modules:\n{1}".format(testtarget,traceback.format_exc()))
+ except AttributeError as e:
+ bb.fatal("Invalid TEST_TARGET - " + str(e))
+ return controller(d, logger)
+
+
class BaseTarget(object, metaclass=ABCMeta):
supported_image_fstypes = []
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v6 10/15] oeqa/targetcontrol: add missing arg to SimpleRemoteTarget.__init__
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
` (8 preceding siblings ...)
2017-11-20 12:15 ` [PATCH v6 09/15] oeqa/targetcontrol: re-introduce get_target_controller() Markus Lehtonen
@ 2017-11-20 12:15 ` Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 11/15] devtools/images: add python-pgo-image Markus Lehtonen
` (4 subsequent siblings)
14 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-20 12:15 UTC (permalink / raw)
To: openembedded-core
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/lib/oeqa/targetcontrol.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/lib/oeqa/targetcontrol.py b/meta/lib/oeqa/targetcontrol.py
index d147a44480..da529e67a4 100644
--- a/meta/lib/oeqa/targetcontrol.py
+++ b/meta/lib/oeqa/targetcontrol.py
@@ -225,8 +225,8 @@ class QemuTarget(BaseTarget):
class SimpleRemoteTarget(BaseTarget):
- def __init__(self, d):
- super(SimpleRemoteTarget, self).__init__(d)
+ def __init__(self, d, logger):
+ super(SimpleRemoteTarget, self).__init__(d, logger)
addr = d.getVar("TEST_TARGET_IP") or bb.fatal('Please set TEST_TARGET_IP with the IP address of the machine you want to run the tests on.')
self.ip = addr.split(":")[0]
try:
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v6 11/15] devtools/images: add python-pgo-image
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
` (9 preceding siblings ...)
2017-11-20 12:15 ` [PATCH v6 10/15] oeqa/targetcontrol: add missing arg to SimpleRemoteTarget.__init__ Markus Lehtonen
@ 2017-11-20 12:15 ` Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 12/15] python3: support profile optimized build Markus Lehtonen
` (3 subsequent siblings)
14 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-20 12:15 UTC (permalink / raw)
To: openembedded-core
This is a special image for profiling Python 3.x in order to utilize
profile-guided-optimization. Profile data can be obtained by by running
bitbake python-pgo-image -c profile. Profile data will be copied into a
directory pointed by PYTHON3_PROFILE_DIR on the host system. The profile
task may be specified with PYTHON3_PROFILE_TASK.
The default profile task is pybench, although an example for using
test.regrtest is included in the recipe.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/images/python-pgo-image.bb | 92 ++++++++++++++++++++++++
1 file changed, 92 insertions(+)
create mode 100644 meta/recipes-devtools/images/python-pgo-image.bb
diff --git a/meta/recipes-devtools/images/python-pgo-image.bb b/meta/recipes-devtools/images/python-pgo-image.bb
new file mode 100644
index 0000000000..0dbe8fd052
--- /dev/null
+++ b/meta/recipes-devtools/images/python-pgo-image.bb
@@ -0,0 +1,92 @@
+SUMMARY = "Minimal image for doing Python profiling (for PGO)"
+
+IMAGE_FEATURES += "ssh-server-dropbear"
+IMAGE_INSTALL = "packagegroup-core-boot"
+IMAGE_INSTALL += "python3-profile-opt python3-profile-opt-tests python3-profile-opt-tools"
+
+LICENSE = "MIT"
+
+inherit core-image
+
+PROFILE_DATA_WORKDIR = "${WORKDIR}/profile-data"
+
+PYTHON3_PROFILE_DIR ?= "${TMPDIR}/work-shared/${MACHINE}/python3/pgo-data"
+PYTHON3_PROFILE_TASK_DEFAULT = "-m test.regrtest --pgo -w -x test_asyncore test_gdb test_multiprocessing_fork test_multiprocessing_forkserver test_multiprocessing_main_handling test_multiprocessing_spawn test_subprocess"
+# Exclude tests that are failing on qemux86
+PYTHON3_PROFILE_TASK_DEFAULT += "test_builtin test_cmath test_concurrent_futures test_difflib test_distutils test_float test_format test_math test_optparse test_shutil test_statistics test_types test_unicode"
+# Exclude tests that are taking very long on qemux86
+PYTHON3_PROFILE_TASK_DEFAULT += "test_lib2to3 test_buffer test_pickle test_io test_threading test_asyncio test_urllib2_localnet test_itertools test_tuple test_trace test_tarfile test_unicodedata test_decimal test_long test_zipfile test_deque test_descr test_email test_venv test_bytes test_compileall test_ast test_multibytecodec"
+
+# Change default profile target to pybench. Running test.regrtest takes
+# ridiculously long, i.e. around 4 hours in qemux86 on i7-3770K desktop
+# machine. Pybench "only" takes around 55 minutes.
+PYTHON3_PROFILE_TASK_DEFAULT = "${docdir}/python3-profile-opt/Tools/pybench/pybench.py -n 2 --with-gc --with-syscheck"
+
+PYTHON3_PROFILE_TASK ?= "${PYTHON3_PROFILE_TASK_DEFAULT}"
+
+# We need these because we're utilizing the runtime test helpers from oeqa
+TEST_TARGET ?= "qemu"
+TEST_QEMUBOOT_TIMEOUT ?= "1000"
+TEST_LOG_DIR ?= "${WORKDIR}/qemulogs"
+FIND_ROOTFS = "1"
+
+def run_profile(d, profile_bin, profile_task, tgt_in_dir, host_out_dir):
+ import logging
+ from oeqa.targetcontrol import get_target_controller
+ from oeqa.utils import make_logger_bitbake_compatible
+ from oe.utils import getstatusoutput
+
+ logger = make_logger_bitbake_compatible(logging.getLogger("TargetControl"))
+ target = get_target_controller(d, logger)
+ target.deploy()
+ try:
+ # Boot target
+ bootparams = None
+ if d.getVar('VIRTUAL-RUNTIME_init_manager', True) == 'systemd':
+ bootparams = 'systemd.log_level=debug systemd.log_target=console'
+ target.start(extra_bootparams=bootparams)
+
+ # Run profile task
+ ret, output = target.run(profile_bin + ' ' + profile_task, timeout=7200)
+ if ret:
+ bb.fatal("Failed to run profile task on target: %s" % output)
+ ret, output = target.run('tar czf pgo-data.tgz -C %s .' % tgt_in_dir)
+ if ret:
+ bb.fatal("Failed to archive profile data on target: %s" % output)
+
+ # Retrieve and unpack profile data
+ target.copy_from('/home/root/pgo-data.tgz', host_out_dir)
+
+ profile_tarball = os.path.join(host_out_dir, 'pgo-data.tgz')
+ ret, output = getstatusoutput('tar xf %s -C %s' % (profile_tarball, host_out_dir))
+ os.unlink(profile_tarball)
+ if ret:
+ bb.fatal("Failed to unpack python profile data: %s" % output)
+ finally:
+ target.stop()
+
+
+# Profile task for Python 3.x
+python do_profile() {
+ run_profile(d, 'python3',
+ d.getVar('PYTHON3_PROFILE_TASK', True),
+ '/home/root/python3-pgo-profiles/',
+ os.path.join(d.getVar("PROFILE_DATA_WORKDIR", True), 'python3'))
+}
+
+addtask profile after do_build
+
+# Strictly, these are dependencies of the do_profile task, but, we cannot put
+# them in task depends because this would pull all possible build deps into the
+# recipe sysroot causing conflicts (python3 and python3-profile-opt)
+DEPENDS += "qemu-native qemu-helper-native openssh-native iproute2-native"
+do_profile[cleandirs] = "${PROFILE_DATA_WORKDIR}/python3"
+
+python do_profile_setscene () {
+ sstate_setscene(d)
+}
+
+SSTATETASKS += "do_profile"
+do_profile[sstate-inputdirs] = "${PROFILE_DATA_WORKDIR}/python3"
+do_profile[sstate-outputdirs] = "${PYTHON3_PROFILE_DIR}"
+addtask do_profile_setscene
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v6 12/15] python3: support profile optimized build
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
` (10 preceding siblings ...)
2017-11-20 12:15 ` [PATCH v6 11/15] devtools/images: add python-pgo-image Markus Lehtonen
@ 2017-11-20 12:15 ` Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 13/15] python3: fix profile-optimized build of modules Markus Lehtonen
` (2 subsequent siblings)
14 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-20 12:15 UTC (permalink / raw)
To: openembedded-core
This patch makes it possible to build python3 with profile directed
optimization. That is, feed python build process with profile data to
guide optimization. This is the third (and the last) step in profile
directed optimization for Python 3.x.
In order to do a profile-optimized build you need to specify
PYTHON3_PROFILE_OPT = "1" in your local.conf, and, have profile data available
in the location pointed to by PYTHON3_PROFILE_DIR. Profile data can be
obtainen e.g. by running bitbake python-pgo-image -c profile3.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python3_3.5.3.bb | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
index bb524eb17d..844139dfa3 100644
--- a/meta/recipes-devtools/python/python3_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3_3.5.3.bb
@@ -79,6 +79,17 @@ export CROSSPYTHONPATH = "${STAGING_LIBDIR_NATIVE}/python${PYTHON_MAJMIN}/lib-dy
# No ctypes option for python 3
PYTHONLSBOPTS = ""
+# Automatic profile guided optimization
+PYTHON3_MAKE_TARGET ?= "${@'build_all_use_profile' if d.getVar('PYTHON3_PROFILE_OPT', True) == '1' else ''}"
+PYTHON3_PROFILE_DIR ?= "${@'${TMPDIR}/work-shared/${MACHINE}/python3/pgo-data' if d.getVar('PYTHON3_PROFILE_OPT', True) == '1' else ''}"
+python () {
+ if (d.getVar('PYTHON3_PROFILE_OPT', True) == '1' and
+ d.getVar('PYTHON3_MAKE_TARGET', True) == 'build_all_use_profile'):
+ profile_dir = d.getVar('PYTHON3_PROFILE_DIR', True)
+ bb.utils.mkdirhier(profile_dir)
+ d.setVarFlag('do_compile', 'file-checksums', '%s:True' % profile_dir)
+}
+
do_configure_append() {
rm -f ${S}/Makefile.orig
autoreconf -Wcross --verbose --install --force --exclude=autopoint ../Python-${PV}/Modules/_ctypes/libffi
@@ -118,6 +129,10 @@ do_compile() {
if [ "${PYTHON3_MAKE_TARGET}" = "build_all_generate_profile" ]; then
# This is only used in PGO profiling by python3-profile-opt package
export EXTRA_CFLAGS="-fprofile-dir=./python3-pgo-profiles/"
+ elif [ -n "${PYTHON3_PROFILE_DIR}" ]; then
+ export EXTRA_CFLAGS="-fprofile-dir=${PYTHON3_PROFILE_DIR}"
+ # Remove non-optimized build artefacts
+ oe_runmake clean
fi
oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
@@ -138,6 +153,11 @@ do_install() {
install -d ${D}${libdir}/pkgconfig
install -d ${D}${libdir}/python${PYTHON_MAJMIN}/config
+ # This only has effect if we build with -fprofile-use, e.g. when make
+ # target is build_all_use_profile
+ if [ -n "${PYTHON3_PROFILE_DIR}" ]; then
+ export EXTRA_CFLAGS="-fprofile-dir=${PYTHON3_PROFILE_DIR}"
+ fi
# rerun the build once again with original makefile this time
# run install in a separate step to avoid compile/install race
oe_runmake HOSTPGEN=${STAGING_BINDIR_NATIVE}/python3-native/pgen \
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v6 13/15] python3: fix profile-optimized build of modules
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
` (11 preceding siblings ...)
2017-11-20 12:15 ` [PATCH v6 12/15] python3: support profile optimized build Markus Lehtonen
@ 2017-11-20 12:15 ` Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 14/15] python3: add python3-tools subpackage Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 15/15] python3: remove two setup.py cross-compile hacks Markus Lehtonen
14 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-20 12:15 UTC (permalink / raw)
To: openembedded-core
Without this the pgo-related compiler flags are not used in
cross-builds.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
...-CFLAGS-for-extensions-when-cross-compili.patch | 56 ++++++++++++++++++++++
meta/recipes-devtools/python/python3_3.5.3.bb | 1 +
2 files changed, 57 insertions(+)
create mode 100644 meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch
diff --git a/meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch b/meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch
new file mode 100644
index 0000000000..2e299b3c59
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/Use-correct-CFLAGS-for-extensions-when-cross-compili.patch
@@ -0,0 +1,56 @@
+From d721ebabbf8567d42dfdcd17e170846ba837b04a Mon Sep 17 00:00:00 2001
+From: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+Date: Wed, 23 Nov 2016 16:08:04 +0200
+Subject: [PATCH] Use correct CFLAGS for extensions when cross-compiling
+
+Take PY_CFLAGS_NODIST into account, like in native build. This is needed
+in order to to profile-optimized build. Also, pass EXTRA_CFLAGS to
+profile-optimized build.
+
+Upstream-Status: Pending
+
+Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
+---
+ Makefile.pre.in | 4 ++--
+ setup.py | 3 ++-
+ 2 files changed, 4 insertions(+), 3 deletions(-)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index 4fcaafc..2d20753 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -510,7 +510,7 @@ profile-opt:
+ $(MAKE) profile-removal
+
+ build_all_generate_profile:
+- $(MAKE) @DEF_MAKE_RULE@ CFLAGS_NODIST="$(CFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LIBS="$(LIBS)"
++ $(MAKE) @DEF_MAKE_RULE@ CFLAGS_NODIST="$(CFLAGS) $(EXTRA_CFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) $(PGO_PROF_GEN_FLAG) @LTOFLAGS@" LIBS="$(LIBS)"
+
+ run_profile_task:
+ : # FIXME: can't run for a cross build
+@@ -520,7 +520,7 @@ build_all_merge_profile:
+ $(LLVM_PROF_MERGER)
+
+ build_all_use_profile:
+- $(MAKE) @DEF_MAKE_RULE@ CFLAGS_NODIST="$(CFLAGS) $(PGO_PROF_USE_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) @LTOFLAGS@"
++ $(MAKE) @DEF_MAKE_RULE@ CFLAGS_NODIST="$(CFLAGS) $(EXTRA_CFLAGS) $(PGO_PROF_USE_FLAG) @LTOFLAGS@" LDFLAGS="$(LDFLAGS) @LTOFLAGS@"
+
+ # Compile and run with gcov
+ .PHONY=coverage coverage-lcov coverage-report
+diff --git a/setup.py b/setup.py
+index 7fa9cc3..ef18fe4 100644
+--- a/setup.py
++++ b/setup.py
+@@ -263,7 +263,8 @@ class PyBuildExt(build_ext):
+ # compilers
+ if compiler is not None:
+ if cross_compiling:
+- (ccshared,cflags) = (os.environ.get('CCSHARED') or '', os.environ.get('CFLAGS') or '')
++ (ccshared,cflags) = (os.environ.get('CCSHARED') or '',
++ (os.environ.get('CFLAGS') or '') + ' ' + sysconfig.get_config_var('PY_CFLAGS_NODIST'))
+ else:
+ (ccshared,cflags) = sysconfig.get_config_vars('CCSHARED','CFLAGS')
+ args['compiler_so'] = compiler + ' ' + ccshared + ' ' + cflags
+--
+2.12.3
+
diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
index 844139dfa3..16097e4236 100644
--- a/meta/recipes-devtools/python/python3_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3_3.5.3.bb
@@ -40,6 +40,7 @@ SRC_URI += "\
file://0001-Issue-21272-Use-_sysconfigdata.py-to-initialize-dist.patch \
file://Fix-29519-weakref-spewing-exceptions-during-interp-f.patch \
file://pass-missing-libraries-to-Extension-for-mul.patch \
+ file://Use-correct-CFLAGS-for-extensions-when-cross-compili.patch \
"
SRC_URI[md5sum] = "57d1f8bfbabf4f2500273fb0706e6f21"
SRC_URI[sha256sum] = "eefe2ad6575855423ab630f5b51a8ef6e5556f774584c06beab4926f930ddbb0"
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v6 14/15] python3: add python3-tools subpackage
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
` (12 preceding siblings ...)
2017-11-20 12:15 ` [PATCH v6 13/15] python3: fix profile-optimized build of modules Markus Lehtonen
@ 2017-11-20 12:15 ` Markus Lehtonen
2017-11-20 23:36 ` Tim Orling
2017-11-20 12:15 ` [PATCH v6 15/15] python3: remove two setup.py cross-compile hacks Markus Lehtonen
14 siblings, 1 reply; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-20 12:15 UTC (permalink / raw)
To: openembedded-core
Useful in developing Python, e.g. in benchmarking.
[YOCTO #9338]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
meta/recipes-devtools/python/python-3.5-manifest.inc | 8 +++++---
meta/recipes-devtools/python/python3_3.5.3.bb | 5 +++++
2 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/meta/recipes-devtools/python/python-3.5-manifest.inc b/meta/recipes-devtools/python/python-3.5-manifest.inc
index 067f6c58c5..3107d48f7e 100644
--- a/meta/recipes-devtools/python/python-3.5-manifest.inc
+++ b/meta/recipes-devtools/python/python-3.5-manifest.inc
@@ -4,9 +4,9 @@
-PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc "
+PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-tools "
-PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-modules"
+PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-modules ${PN}-tools"
SUMMARY_${PN}-2to3="Python automated Python 2 to 3 code translator"
RDEPENDS_${PN}-2to3="${PN}-core"
@@ -280,4 +280,6 @@ SUMMARY_${PN}-modules="All Python modules"
RDEPENDS_${PN}-modules="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-difflib ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc "
ALLOW_EMPTY_${PN}-modules = "1"
-
+SUMMARY_${PN}-tools="Useful scripts from Python"
+RDEPENDS_${PN}-tools="${PN}-modules"
+FILES_${PN}-tools="${docdir}/${PN}/Tools"
diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
index 16097e4236..7746ffe3b1 100644
--- a/meta/recipes-devtools/python/python3_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3_3.5.3.bb
@@ -196,6 +196,11 @@ do_install() {
fi
oe_multilib_header python${PYTHON_BINABI}/pyconfig.h
+
+ # Install tools
+ install -d -m 755 ${D}/${docdir}/${PN}
+ cp -r ${S}/Tools ${D}/${docdir}/${PN}/
+ find ${D}/${docdir}/${PN}/ -name Makefile -delete
}
do_install_append_class-nativesdk () {
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* [PATCH v6 15/15] python3: remove two setup.py cross-compile hacks
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
` (13 preceding siblings ...)
2017-11-20 12:15 ` [PATCH v6 14/15] python3: add python3-tools subpackage Markus Lehtonen
@ 2017-11-20 12:15 ` Markus Lehtonen
14 siblings, 0 replies; 41+ messages in thread
From: Markus Lehtonen @ 2017-11-20 12:15 UTC (permalink / raw)
To: openembedded-core
Remove two unneeded hacks. The first hack ("setup.py: no host headers
libs" patch) is not needed because we use cross-compiler (e.g.
i586-oe-linux-gcc) which has not been configured with any host system
include or library directories, and thus, we don't get any host system
directories when running "gcc -E -v".
The second hack becomes useless after the first hack has been removed
and we get the standard include and lib directories normally from gcc.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
| 33 ----------------------
.../setup.py-find-libraries-in-staging-dirs.patch | 30 --------------------
meta/recipes-devtools/python/python3_3.5.3.bb | 2 --
3 files changed, 65 deletions(-)
delete mode 100644 meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch
delete mode 100644 meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
diff --git a/meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch b/meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch
deleted file mode 100644
index 2bc8b8c460..0000000000
--- a/meta/recipes-devtools/python/python3/python3-setup.py-no-host-headers-libs.patch
+++ /dev/null
@@ -1,33 +0,0 @@
-From 43238e1ac13e32984d015c92a5841f3de1fe1d15 Mon Sep 17 00:00:00 2001
-From: Jackie Huang <jackie.huang@windriver.com>
-Date: Tue, 18 Nov 2014 00:07:07 -0500
-Subject: [PATCH] setup.py: no host headers libs
-
-When we are cross-compiling, setup.py should never look in /usr
-or /usr/local to find headers or libraries.
-
-Upstream-Status: Inappropriate [Cross compile specific]
-
-Signed-off-by: Jackie Huang <jackie.huang@windriver.com>
----
- setup.py | 3 ---
- 1 file changed, 3 deletions(-)
-
-diff --git a/setup.py b/setup.py
-index f020b28..e8339cd 100644
---- a/setup.py
-+++ b/setup.py
-@@ -444,10 +444,7 @@ class PyBuildExt(build_ext):
- if not cross_compiling:
- add_dir_to_list(self.compiler.library_dirs, os.path.join('/usr/local', sys.lib))
- add_dir_to_list(self.compiler.include_dirs, '/usr/local/include')
-- # only change this for cross builds for 3.3, issues on Mageia
-- if cross_compiling:
- self.add_gcc_paths()
-- if not cross_compiling:
- self.add_multiarch_paths()
-
- # Add paths specified in the environment variables LDFLAGS and
---
-2.0.0
-
diff --git a/meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch b/meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
deleted file mode 100644
index f26bd38b02..0000000000
--- a/meta/recipes-devtools/python/python3/setup.py-find-libraries-in-staging-dirs.patch
+++ /dev/null
@@ -1,30 +0,0 @@
-Patch setup.py so that the detect_modules() function looks for required
-libraries and headers in STAGING_LIBDIR / STAGING_INCDIR.
-
-Without this patch, several extension modules are not built, even though
-their dependencies are present in the compiler's search paths.
-The result is the following warning, and ultimately incomplete packages:
-
-| The necessary bits to build these optional modules were not found:
-| _bz2 _curses_panel _dbm
-| _gdbm _lzma _sqlite3
-| nis readline zlib
-| To find the necessary bits, look in setup.py in detect_modules() for the module's name.
-
-Upstream-Status: Inappropriate [Cross compile specific]
-
-Signed-off-by: Dominic Sacré <dominic.sacre@gmx.de>
-
-Index: Python-3.4.3/setup.py
-===================================================================
---- Python-3.4.3.orig/setup.py
-+++ Python-3.4.3/setup.py
-@@ -521,6 +521,8 @@ class PyBuildExt(build_ext):
- else:
- lib_dirs = self.compiler.library_dirs[:]
- inc_dirs = self.compiler.include_dirs[:]
-+ lib_dirs.append(os.environ.get('STAGING_LIBDIR'))
-+ inc_dirs.append(os.environ.get('STAGING_INCDIR'))
- exts = []
- missing = []
-
diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
index 7746ffe3b1..3a997a3b4e 100644
--- a/meta/recipes-devtools/python/python3_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3_3.5.3.bb
@@ -31,10 +31,8 @@ SRC_URI += "\
file://unixccompiler.patch \
file://avoid-ncursesw-include-path.patch \
file://python3-use-CROSSPYTHONPATH-for-PYTHON_FOR_BUILD.patch \
- file://python3-setup.py-no-host-headers-libs.patch \
file://sysconfig.py-add-_PYTHON_PROJECT_SRC.patch \
file://setup.py-check-cross_compiling-when-get-FLAGS.patch \
- file://setup.py-find-libraries-in-staging-dirs.patch \
file://configure.ac-fix-LIBPL.patch \
file://upstream-random-fixes.patch \
file://0001-Issue-21272-Use-_sysconfigdata.py-to-initialize-dist.patch \
--
2.13.6
^ permalink raw reply related [flat|nested] 41+ messages in thread
* ✗ patchtest: failure for support profile-optimized build for Python (rev2)
2017-10-27 10:07 [PATCH v5 00/10] support profile-optimized build for Python Markus Lehtonen
` (11 preceding siblings ...)
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
@ 2017-11-20 12:33 ` Patchwork
12 siblings, 0 replies; 41+ messages in thread
From: Patchwork @ 2017-11-20 12:33 UTC (permalink / raw)
To: Markus Lehtonen; +Cc: openembedded-core
== Series Details ==
Series: support profile-optimized build for Python (rev2)
Revision: 2
URL : https://patchwork.openembedded.org/series/9511/
State : failure
== Summary ==
Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:
* Issue Added patch file is missing Upstream-Status in the header [test_upstream_status_presence_format]
Suggested fix Add Upstream-Status: <Valid status> to the header of meta/recipes-connectivity/openssh/openssh/0001-don-t-use-absolute-path-for-ssh-program.patch
Standard format Upstream-Status: <Valid status>
Valid status Pending, Accepted, Backport, Denied, Inappropriate [reason], Submitted [where]
If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).
---
Guidelines: https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v6 14/15] python3: add python3-tools subpackage
2017-11-20 12:15 ` [PATCH v6 14/15] python3: add python3-tools subpackage Markus Lehtonen
@ 2017-11-20 23:36 ` Tim Orling
0 siblings, 0 replies; 41+ messages in thread
From: Tim Orling @ 2017-11-20 23:36 UTC (permalink / raw)
To: Markus Lehtonen; +Cc: openembedded-core
The top of the manifest file says that it is auto generated [1], this change must also be done in scripts/contrib/python/generate-manifest-3.5.py [2]:
[1] http://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/python/python-3.5-manifest.inc#n1
[2] http://git.openembedded.org/openembedded-core/tree/scripts/contrib/python/generate-manifest-3.5.py
Nacked-by: Tim Orling <timothy.t.orling@linux.intel.com>
> On Nov 20, 2017, at 4:15 AM, Markus Lehtonen <markus.lehtonen@linux.intel.com> wrote:
>
> Useful in developing Python, e.g. in benchmarking.
>
> [YOCTO #9338]
>
> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
> ---
> meta/recipes-devtools/python/python-3.5-manifest.inc | 8 +++++---
> meta/recipes-devtools/python/python3_3.5.3.bb | 5 +++++
> 2 files changed, 10 insertions(+), 3 deletions(-)
>
> diff --git a/meta/recipes-devtools/python/python-3.5-manifest.inc b/meta/recipes-devtools/python/python-3.5-manifest.inc
> index 067f6c58c5..3107d48f7e 100644
> --- a/meta/recipes-devtools/python/python-3.5-manifest.inc
> +++ b/meta/recipes-devtools/python/python-3.5-manifest.inc
> @@ -4,9 +4,9 @@
>
>
>
> -PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc "
> +PROVIDES+="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils ${PN}-distutils-staticdev ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-tools "
>
> -PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-modules"
> +PACKAGES="${PN}-dbg ${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-dev ${PN}-difflib ${PN}-distutils-staticdev ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-tests ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc ${PN}-modules ${PN}-tools"
>
> SUMMARY_${PN}-2to3="Python automated Python 2 to 3 code translator"
> RDEPENDS_${PN}-2to3="${PN}-core"
> @@ -280,4 +280,6 @@ SUMMARY_${PN}-modules="All Python modules"
> RDEPENDS_${PN}-modules="${PN}-2to3 ${PN}-argparse ${PN}-asyncio ${PN}-audio ${PN}-codecs ${PN}-compile ${PN}-compression ${PN}-core ${PN}-crypt ${PN}-ctypes ${PN}-curses ${PN}-datetime ${PN}-db ${PN}-debugger ${PN}-difflib ${PN}-distutils ${PN}-doctest ${PN}-email ${PN}-enum ${PN}-fcntl ${PN}-gdbm ${PN}-html ${PN}-idle ${PN}-image ${PN}-importlib ${PN}-io ${PN}-json ${PN}-lang ${PN}-logging ${PN}-mailbox ${PN}-math ${PN}-mime ${PN}-mmap ${PN}-multiprocessing ${PN}-netclient ${PN}-netserver ${PN}-numbers ${PN}-pickle ${PN}-pkgutil ${PN}-pprint ${PN}-profile ${PN}-pydoc ${PN}-re ${PN}-readline ${PN}-reprlib ${PN}-resource ${PN}-selectors ${PN}-shell ${PN}-signal ${PN}-smtpd ${PN}-sqlite3 ${PN}-sqlite3-tests ${PN}-stringold ${PN}-subprocess ${PN}-syslog ${PN}-terminal ${PN}-textutils ${PN}-threading ${PN}-tkinter ${PN}-typing ${PN}-unittest ${PN}-unixadmin ${PN}-xml ${PN}-xmlrpc "
> ALLOW_EMPTY_${PN}-modules = "1"
>
> -
> +SUMMARY_${PN}-tools="Useful scripts from Python"
> +RDEPENDS_${PN}-tools="${PN}-modules"
> +FILES_${PN}-tools="${docdir}/${PN}/Tools"
> diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
> index 16097e4236..7746ffe3b1 100644
> --- a/meta/recipes-devtools/python/python3_3.5.3.bb
> +++ b/meta/recipes-devtools/python/python3_3.5.3.bb
> @@ -196,6 +196,11 @@ do_install() {
> fi
>
> oe_multilib_header python${PYTHON_BINABI}/pyconfig.h
> +
> + # Install tools
> + install -d -m 755 ${D}/${docdir}/${PN}
> + cp -r ${S}/Tools ${D}/${docdir}/${PN}/
> + find ${D}/${docdir}/${PN}/ -name Makefile -delete
> }
>
> do_install_append_class-nativesdk () {
> --
> 2.13.6
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 41+ messages in thread
* Re: [PATCH v6 02/15] python3: fix depends of python3-tests
2017-11-20 12:15 ` [PATCH v6 02/15] python3: fix depends of python3-tests Markus Lehtonen
@ 2017-11-20 23:37 ` Tim Orling
0 siblings, 0 replies; 41+ messages in thread
From: Tim Orling @ 2017-11-20 23:37 UTC (permalink / raw)
To: Markus Lehtonen; +Cc: openembedded-core
The top of the manifest file says that it is auto generated [1], this change must also be done in scripts/contrib/python/generate-manifest-3.5.py [2]:
[1] http://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/python/python-3.5-manifest.inc#n1
[2] http://git.openembedded.org/openembedded-core/tree/scripts/contrib/python/generate-manifest-3.5.py
Nacked-by: Tim Orling <timothy.t.orling@linux.intel.com>
> On Nov 20, 2017, at 4:15 AM, Markus Lehtonen <markus.lehtonen@linux.intel.com> wrote:
>
> Similar to an earlier fix for Python 2.7. Make the tests subpackage
> depend on all modules as test.regrtest uses most (if not all) of them.
>
> Some tests also depend on libgcc so add that as a runtime dependency as
> well.
>
> [YOCTO #9338]
>
> Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
> ---
> meta/recipes-devtools/python/python-3.5-manifest.inc | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/recipes-devtools/python/python-3.5-manifest.inc b/meta/recipes-devtools/python/python-3.5-manifest.inc
> index 0260e87e75..067f6c58c5 100644
> --- a/meta/recipes-devtools/python/python-3.5-manifest.inc
> +++ b/meta/recipes-devtools/python/python-3.5-manifest.inc
> @@ -241,7 +241,7 @@ RDEPENDS_${PN}-terminal="${PN}-core ${PN}-io"
> FILES_${PN}-terminal="${libdir}/python3.5/pty.* ${libdir}/python3.5/__pycache__/pty.* ${libdir}/python3.5/tty.* ${libdir}/python3.5/__pycache__/tty.* "
>
> SUMMARY_${PN}-tests="Python tests"
> -RDEPENDS_${PN}-tests="${PN}-core ${PN}-compression"
> +RDEPENDS_${PN}-tests="${PN}-core ${PN}-modules libgcc"
> FILES_${PN}-tests="${libdir}/python3.5/test ${libdir}/python3.5/test/__pycache__ "
>
> SUMMARY_${PN}-textutils="Python option parsing, text wrapping and CSV support"
> --
> 2.13.6
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 41+ messages in thread
end of thread, other threads:[~2017-11-20 23:37 UTC | newest]
Thread overview: 41+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-10-27 10:07 [PATCH v5 00/10] support profile-optimized build for Python Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 01/10] python3-native: support profile optimized build Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 02/10] python3: fix depends of python3-tests Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 03/10] python3: add python3-profile-opt recipe Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 04/10] python3-profile-opt: rename libpython3 Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 05/10] bitbake.conf: add ip and scp to HOSTTOOLS unconditionally Markus Lehtonen
2017-10-30 1:30 ` ChenQi
2017-10-30 10:51 ` Markus Lehtonen
2017-10-30 13:55 ` Konrad Scherer
2017-10-30 14:42 ` Markus Lehtonen
2017-10-31 6:34 ` ChenQi
2017-10-31 7:15 ` Markus Lehtonen
2017-11-01 15:35 ` Konrad Scherer
2017-11-15 12:15 ` Markus Lehtonen
2017-11-16 0:13 ` Andre McCurdy
2017-10-27 10:07 ` [PATCH v5 06/10] devtools/images: add python-pgo-image Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 07/10] python3: support profile optimized build Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 08/10] python3: fix profile-optimized build of modules Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 09/10] python3: add python3-tools subpackage Markus Lehtonen
2017-10-27 10:07 ` [PATCH v5 10/10] python3: remove two setup.py cross-compile hacks Markus Lehtonen
2017-10-27 10:34 ` ✗ patchtest: failure for support profile-optimized build for Python Patchwork
2017-10-27 15:55 ` Daniela Plascencia
2017-11-20 12:15 ` [PATCH v6 00/15] " Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 01/15] python3-native: support profile optimized build Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 02/15] python3: fix depends of python3-tests Markus Lehtonen
2017-11-20 23:37 ` Tim Orling
2017-11-20 12:15 ` [PATCH v6 03/15] python3: add python3-profile-opt recipe Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 04/15] python3-profile-opt: rename libpython3 Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 05/15] openssh: extend to -native Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 06/15] libmnl: enable native Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 07/15] iptables: " Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 08/15] iproute2: " Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 09/15] oeqa/targetcontrol: re-introduce get_target_controller() Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 10/15] oeqa/targetcontrol: add missing arg to SimpleRemoteTarget.__init__ Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 11/15] devtools/images: add python-pgo-image Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 12/15] python3: support profile optimized build Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 13/15] python3: fix profile-optimized build of modules Markus Lehtonen
2017-11-20 12:15 ` [PATCH v6 14/15] python3: add python3-tools subpackage Markus Lehtonen
2017-11-20 23:36 ` Tim Orling
2017-11-20 12:15 ` [PATCH v6 15/15] python3: remove two setup.py cross-compile hacks Markus Lehtonen
2017-11-20 12:33 ` ✗ patchtest: failure for support profile-optimized build for Python (rev2) Patchwork
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox