[PATCH 0/3] Fixes for python3 and dbus-ptest

[PATCH 0/3] Fixes for python3 and dbus-ptest

[kai.kang]

From: Kai Kang <kai.kang@windriver.com>

1
when multilib is enabled, python3 fails to start:
root@qemux86-64:~# python3 
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'

Current thread 0x00007f4d9b268700 (most recent call first):
Aborted

It is library path issue, and fix it. Then check sys.path are right:

root@qemux86-64:~# python3
Python 3.5.2 (default, Nov  1 2016, 15:32:42) 
[GCC 6.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib64/python35.zip', '/usr/lib64/python3.5', '/usr/lib64/python3.5/plat-linux', '/usr/lib64/python3.5/lib-dynload']
>>>


2
Only 4 cases PASS of dbus-ptest. With the fixes, all ptests cases PASS on qemux86-64.
And for qemuarm, only one case fails:

./run-ptest: line 10:   283 Aborted                 ./$i ./test/data DBUS_TEST_HOMEDIR=./test > /dev/null
FAIL: test/test-refs


It is a multi-threads issue, and need more work to fix it.


---
The following changes since commit 003b053498aea91fdb8c5418d0d8bbda7b23dbc0:

  boost: fix the SRC_URI to point to an actual release, and not a master snapshot (2016-10-31 14:23:12 +0000)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib kangkai/python3_and_dbus
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kangkai/python3_and_dbus

Kai Kang (3):
  dbus-test: correct paths in service and conf files
  dbus: update build options when enable ptest
  python3: fix library search path

 meta/recipes-core/dbus/dbus-test_1.10.10.bb        |  3 ++
 meta/recipes-core/dbus/dbus_1.10.10.bb             |  4 +-
 ...t-python3-lib-pathes-for-multilib-support.patch | 49 ++++++++++++++++++++++
 .../python/python3/python-3.3-multilib.patch       | 13 +++---
 meta/recipes-devtools/python/python3_3.5.2.bb      |  1 +
 5 files changed, 61 insertions(+), 9 deletions(-)
 create mode 100644 meta/recipes-devtools/python/python3/correct-python3-lib-pathes-for-multilib-support.patch

-- 
2.10.1

[PATCH 1/3] dbus-test: correct paths in service and conf files

[kai.kang]

From: Kai Kang <kai.kang@windriver.com>

Build path ${B} exists in some service and conf files of ptest. Replace
with right directory.

Signed-off-by: Kai Kang <kai.kang@windriver.com>
---
 meta/recipes-core/dbus/dbus-test_1.10.10.bb | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-core/dbus/dbus-test_1.10.10.bb b/meta/recipes-core/dbus/dbus-test_1.10.10.bb
index 650b7ab..29d7f91 100644
--- a/meta/recipes-core/dbus/dbus-test_1.10.10.bb
+++ b/meta/recipes-core/dbus/dbus-test_1.10.10.bb
@@ -54,5 +54,8 @@ do_install_ptest() {
 	for i in $l; do install ${B}/bus/.libs/test-$i ${D}${PTEST_PATH}/test; done
 	install ${B}/dbus/.libs/test-dbus ${D}${PTEST_PATH}/test
 	cp -r ${B}/test/data ${D}${PTEST_PATH}/test
+
+	# set right path in service and conf files
+	find ${D} -type f -not -executable -exec sed -i 's#${B}#${PTEST_PATH}#' {} \;
 }
 RDEPENDS_${PN}-ptest += "bash"
-- 
2.10.1

[PATCH 2/3] dbus: update build options when enable ptest

[kai.kang]

From: Kai Kang <kai.kang@windriver.com>

It fails to run some dbus-ptest cases that some symbols cannot be found:

| ./test/test-bus: relocation error: ./test/test-bus: symbol
| _dbus_threads_init_debug, version LIBDBUS_PRIVATE_1.10.10 not defined
| in file libdbus-1.so.3 with link time reference
| FAIL: test/test-bus

These missing symbols are controlled by some macros. Update configure
options and compile macro to make the symbols visible to ptest cases.

Signed-off-by: Kai Kang <kai.kang@windriver.com>
---
 meta/recipes-core/dbus/dbus_1.10.10.bb | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-core/dbus/dbus_1.10.10.bb b/meta/recipes-core/dbus/dbus_1.10.10.bb
index 4db0b9b..214cc13 100644
--- a/meta/recipes-core/dbus/dbus_1.10.10.bb
+++ b/meta/recipes-core/dbus/dbus_1.10.10.bb
@@ -90,7 +90,7 @@ pkg_postinst_dbus() {
 	fi
 }
 
-EXTRA_OECONF = "--disable-tests \
+EXTRA_OECONF = "${@bb.utils.contains('DISTRO_FEATURES', 'ptest', '--enable-embedded-tests --enable-asserts --enable-verbose-mode', '--disable-tests', d)} \
                 --disable-xml-docs \
                 --disable-doxygen-docs \
                 --disable-libaudit \
@@ -98,6 +98,8 @@ EXTRA_OECONF = "--disable-tests \
 
 EXTRA_OECONF_append_class-native = " --disable-selinux"
 
+EXTRA_OEMAKE = "${@bb.utils.contains('DISTRO_FEATURES', 'ptest', 'CFLAG_VISIBILITY=-fvisibility=default', '', d)}"
+
 PACKAGECONFIG ??= "${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'systemd', '', d)} \
                    ${@bb.utils.contains('DISTRO_FEATURES', 'largefile', 'largefile', '', d)} \
                    ${@bb.utils.contains('DISTRO_FEATURES', 'x11', 'x11', '', d)}"
-- 
2.10.1

[PATCH 3/3] python3: fix library search path

[kai.kang]

From: Kai Kang <kai.kang@windriver.com>

When enable multilib, it fails to run python3:

| Could not find platform independent libraries <prefix>
| Could not find platform dependent libraries <exec_prefix>
| Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
| Fatal Python error: Py_Initialize: Unable to get the locale encoding
| ImportError: No module named 'encodings'
|
| Current thread 0x00007f62ea5b2700 (most recent call first):
| Aborted (core dumped)

The root cause is the module search path /usr/lib is wrong for x86-64
when multilib is enabled. So replace fixed path string with right macro.

Clean up python-3.3-multilib.patch for Modules/getpath.c at same time.

Signed-off-by: Kai Kang <kai.kang@windriver.com>
---
 ...t-python3-lib-pathes-for-multilib-support.patch | 49 ++++++++++++++++++++++
 .../python/python3/python-3.3-multilib.patch       | 13 +++---
 meta/recipes-devtools/python/python3_3.5.2.bb      |  1 +
 3 files changed, 55 insertions(+), 8 deletions(-)
 create mode 100644 meta/recipes-devtools/python/python3/correct-python3-lib-pathes-for-multilib-support.patch

diff --git a/meta/recipes-devtools/python/python3/correct-python3-lib-pathes-for-multilib-support.patch b/meta/recipes-devtools/python/python3/correct-python3-lib-pathes-for-multilib-support.patch
new file mode 100644
index 0000000..8ca5a6a
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/correct-python3-lib-pathes-for-multilib-support.patch
@@ -0,0 +1,49 @@
+When enable multilib, it fails to run python3:
+
+| Could not find platform independent libraries <prefix>
+| Could not find platform dependent libraries <exec_prefix>
+| Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
+| Fatal Python error: Py_Initialize: Unable to get the locale encoding
+| ImportError: No module named 'encodings'
+|
+| Current thread 0x00007f62ea5b2700 (most recent call first):
+| Aborted (core dumped)
+
+The root cause is the module search path /usr/lib is wrong for x86-64 when
+multilib is enabled. So replace fixed path string with right macro.
+
+Upstream-Status: Pending
+
+Signed-off-by: Kai Kang <kai.kang@windriver.com>
+---
+diff --git a/Modules/getpath.c b/Modules/getpath.c
+index 18deb60..d22af53 100644
+--- a/Modules/getpath.c
++++ b/Modules/getpath.c
+@@ -494,7 +494,7 @@ calculate_path(void)
+     _pythonpath = Py_DecodeLocale(PYTHONPATH, NULL);
+     _prefix = Py_DecodeLocale(PREFIX, NULL);
+     _exec_prefix = Py_DecodeLocale(EXEC_PREFIX, NULL);
+-    lib_python = Py_DecodeLocale("lib/python" VERSION, NULL);
++    lib_python = Py_DecodeLocale(LIB_PYTHON, NULL);
+ 
+     if (!_pythonpath || !_prefix || !_exec_prefix || !lib_python) {
+         Py_FatalError(
+@@ -683,7 +683,7 @@ calculate_path(void)
+     }
+     else
+         wcsncpy(zip_path, _prefix, MAXPATHLEN);
+-    joinpath(zip_path, L"lib/python00.zip");
++    joinpath(zip_path, LIB L"/python00.zip");
+     bufsz = wcslen(zip_path);   /* Replace "00" with version */
+     zip_path[bufsz - 6] = VERSION[0];
+     zip_path[bufsz - 5] = VERSION[2];
+@@ -695,7 +695,7 @@ calculate_path(void)
+             fprintf(stderr,
+                 "Could not find platform dependent libraries <exec_prefix>\n");
+         wcsncpy(exec_prefix, _exec_prefix, MAXPATHLEN);
+-        joinpath(exec_prefix, L"lib/lib-dynload");
++        joinpath(exec_prefix, LIB L"/lib-dynload");
+     }
+     /* If we found EXEC_PREFIX do *not* reduce it!  (Yet.) */
+ 
diff --git a/meta/recipes-devtools/python/python3/python-3.3-multilib.patch b/meta/recipes-devtools/python/python3/python-3.3-multilib.patch
index 056e8e7..f7e73ee 100644
--- a/meta/recipes-devtools/python/python3/python-3.3-multilib.patch
+++ b/meta/recipes-devtools/python/python3/python-3.3-multilib.patch
@@ -7,6 +7,11 @@ get the sys.lib from python itself and do not use hardcoded value of 'lib'
 Signed-off-by: Khem Raj <raj.khem@gmail.com>
 Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
 
+Remove the section to add a global var in Modules/getpath.c that the global var
+has been removed from python3 upstream.
+
+Signed-off-by: Kai Kang <kai.kang@windriver.com>
+
 Index: Python-3.5.2/Include/pythonrun.h
 ===================================================================
 --- Python-3.5.2.orig/Include/pythonrun.h
@@ -130,14 +135,6 @@ Index: Python-3.5.2/Modules/getpath.c
  #ifndef LANDMARK
  #define LANDMARK L"os.py"
  #endif
-@@ -113,6 +120,7 @@ static wchar_t prefix[MAXPATHLEN+1];
- static wchar_t exec_prefix[MAXPATHLEN+1];
- static wchar_t progpath[MAXPATHLEN+1];
- static wchar_t *module_search_path = NULL;
-+static wchar_t *lib_python = L"" LIB_PYTHON;
- 
- /* Get file status. Encode the path to the locale encoding. */
- 
 Index: Python-3.5.2/Python/getplatform.c
 ===================================================================
 --- Python-3.5.2.orig/Python/getplatform.c
diff --git a/meta/recipes-devtools/python/python3_3.5.2.bb b/meta/recipes-devtools/python/python3_3.5.2.bb
index e6cbb9c..7e6f019 100644
--- a/meta/recipes-devtools/python/python3_3.5.2.bb
+++ b/meta/recipes-devtools/python/python3_3.5.2.bb
@@ -37,6 +37,7 @@ SRC_URI += "\
             file://setup.py-find-libraries-in-staging-dirs.patch \
             file://configure.ac-fix-LIBPL.patch \
             file://python3-fix-CVE-2016-1000110.patch \
+            file://correct-python3-lib-pathes-for-multilib-support.patch \
            "
 SRC_URI[md5sum] = "8906efbacfcdc7c3c9198aeefafd159e"
 SRC_URI[sha256sum] = "0010f56100b9b74259ebcd5d4b295a32324b58b517403a10d1a2aa7cb22bca40"
-- 
2.10.1