* [PATCH 2/3] oeqa/selftest: Add test for shebang overflow
2022-06-06 15:58 [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
@ 2022-06-06 15:58 ` Paulo Neves
2022-06-06 15:58 ` [PATCH 3/3] oeqa/selftest: Test staged .la and .pc files Paulo Neves
` (6 subsequent siblings)
7 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-06 15:58 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
Make sure we do not stage any executable with a bigger shebang
than 128. Fixes [1]
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
---
.../sysroot-test/sysroot-shebang-test_1.0.bb | 12 ++++++++++++
meta/lib/oeqa/selftest/cases/sysroot.py | 10 ++++++++++
2 files changed, 22 insertions(+)
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
new file mode 100644
index 0000000000..6c834be897
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+ install -d ${D}${bindir}
+ echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+ chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 315d1a61c2..9457f1e3ac 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -35,3 +35,13 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
TESTSTRING:pn-sysroot-test-arch2 = "%s"
""" % (uuid1, uuid2))
bitbake("sysroot-test")
+
+ def test_sysroot_max_shebang(self):
+ """
+ Summary: Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+ Expected: Fail when a shebang bigger than the max shebang-size is reached.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+ res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH 3/3] oeqa/selftest: Test staged .la and .pc files
2022-06-06 15:58 [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
2022-06-06 15:58 ` [PATCH 2/3] oeqa/selftest: Add test for shebang overflow Paulo Neves
@ 2022-06-06 15:58 ` Paulo Neves
2022-06-07 15:11 ` [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py Paulo Neves
` (5 subsequent siblings)
7 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-06 15:58 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
These files are checked by qa_check_staged but there was no
test cases for whether the tests actually worked. Now there
are.
---
.../sysroot-test/sysroot-la-test_1.0.bb | 16 ++++++++++
.../sysroot-test/sysroot-pc-test_1.0.bb | 12 +++++++
meta/lib/oeqa/selftest/cases/sysroot.py | 32 +++++++++++++++++++
3 files changed, 60 insertions(+)
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
new file mode 100644
index 0000000000..21f06782fb
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+ install -d ${D}${libdir}/test/
+ echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
new file mode 100644
index 0000000000..e748310fc4
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+ install -d ${D}${libdir}/test/
+ echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 9457f1e3ac..588fc8c713 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -45,3 +45,35 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
self.assertTrue(expected in res.output, msg=res.output)
+
+ def test_sysroot_la(self):
+ """
+ Summary: Check that workdir paths are not contained in .la files.
+ Expected: Fail when a workdir path is found in the file content.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "la-test.la failed sanity test (workdir) in path"
+
+ res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[la]' in res.output, msg=res.output)
+
+ res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[la]' in res.output, msg=res.output)
+
+ def test_sysroot_pkgconfig(self):
+ """
+ Summary: Check that tmpdir paths are not contained in .pc files.
+ Expected: Fail when a tmpdir path is found in the file content.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "test.pc failed sanity test (tmpdir) in path"
+
+ res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+ self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+ self.assertTrue(expected in res.output, msg=res.output)
+
+ res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py
2022-06-06 15:58 [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
2022-06-06 15:58 ` [PATCH 2/3] oeqa/selftest: Add test for shebang overflow Paulo Neves
2022-06-06 15:58 ` [PATCH 3/3] oeqa/selftest: Test staged .la and .pc files Paulo Neves
@ 2022-06-07 15:11 ` Paulo Neves
2022-06-07 15:11 ` [PATCH v2 2/5] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2 Paulo Neves
` (4 more replies)
2022-06-10 21:43 ` [PATCH v3 1/7] " Paulo Neves
` (4 subsequent siblings)
7 siblings, 5 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-07 15:11 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
The native path may be too big, leading to shebang
overflow. Just use the #!/usr/bin/env python3.
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
.../recipes-devtools/python/python3_3.10.4.bb | 1 +
2 files changed, 30 insertions(+)
create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
new file mode 100644
index 0000000000..9f23278a30
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
@@ -0,0 +1,29 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <ptsneves@gmail.com>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Upstream-Status: Inappropriate [distribution]
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+ @ # Substitution happens here, as the completely-expanded BINDIR
+ @ # is not available in configure
+ sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++ @ # Otherwise we might get huge shebangs with native paths
++ sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+ @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+ LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+ @ # In OpenEmbedded, always use the python version of the script, the shell
+--
+2.25.1
+
diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
index 6bd3a6aba8..357025f856 100644
--- a/meta/recipes-devtools/python/python3_3.10.4.bb
+++ b/meta/recipes-devtools/python/python3_3.10.4.bb
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
file://deterministic_imports.patch \
+ file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
"
SRC_URI:append:class-native = " \
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v2 2/5] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2
2022-06-07 15:11 ` [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py Paulo Neves
@ 2022-06-07 15:11 ` Paulo Neves
2022-06-07 15:11 ` [PATCH v2 3/5] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
` (3 subsequent siblings)
4 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-07 15:11 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
A native path can cause a shebang overflow on gtkdoc-mkhtml.
Replace it with /usr/bin/env.
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
index 150d2c0b23..392913fcc6 100644
--- a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
+++ b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
@@ -43,6 +43,7 @@ do_install:append () {
${datadir}/gtk-doc/python/gtkdoc/config.py; do
sed -e 's,${RECIPE_SYSROOT_NATIVE}/usr/bin/pkg-config,${bindir}/pkg-config,' \
-e 's,${HOSTTOOLS_DIR}/python3,${bindir}/python3,' \
+ -e '1s|^#!.*|#!/usr/bin/env python3|' \
-i ${D}$fn
done
}
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v2 3/5] insane.bbclass: Make do_qa_staging check shebangs
2022-06-07 15:11 ` [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py Paulo Neves
2022-06-07 15:11 ` [PATCH v2 2/5] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2 Paulo Neves
@ 2022-06-07 15:11 ` Paulo Neves
2022-06-07 15:11 ` [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow Paulo Neves
` (2 subsequent siblings)
4 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-07 15:11 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
As reported in the bug report [1], there was no check for shebang
sizes on native scripts and now this is fixed.
The path scope of the qa_staging was increased from just checking
libdir to all the relevant SYSROOT_DIRS.
It is possible to skip this check through INSANE_SKIP.
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
meta/classes/insane.bbclass | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 9ca84bace9..b2951a48fe 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
skip_pkgconfig = True
+ skip_shebang_size = False
+ if 'shebang-size' in skip:
+ bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+ skip_shebang_size = True
+
# find all .la and .pc files
# read the content
# and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
oe.qa.handle_error("pkgconfig", error_msg, d)
+ if not skip_shebang_size:
+ errors = {}
+ package_qa_check_shebang_size(path, "", d, None, errors)
+ for e in errors:
+ oe.qa.handle_error(e, errors[e], d)
+
+
# Run all package-wide warnfuncs and errorfuncs
def package_qa_package(warnfuncs, errorfuncs, package, d):
warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
python do_qa_staging() {
bb.note("QA checking staging")
- qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+ sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+ for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+ qa_check_staged(sysroot_destdir + sysroot_dir, d)
oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
}
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow
2022-06-07 15:11 ` [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py Paulo Neves
2022-06-07 15:11 ` [PATCH v2 2/5] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2 Paulo Neves
2022-06-07 15:11 ` [PATCH v2 3/5] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
@ 2022-06-07 15:11 ` Paulo Neves
2022-06-08 12:53 ` [OE-core] " Luca Ceresoli
` (2 more replies)
2022-06-07 15:11 ` [PATCH v2 5/5] oeqa/selftest: Test staged .la and .pc files Paulo Neves
2022-06-07 16:53 ` [OE-core] [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py Alexander Kanavin
4 siblings, 3 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-07 15:11 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
Make sure we do not stage any executable with a bigger shebang
than 128. Fixes [1]
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
.../sysroot-test/sysroot-shebang-test_1.0.bb | 12 ++++++++++++
meta/lib/oeqa/selftest/cases/sysroot.py | 10 ++++++++++
2 files changed, 22 insertions(+)
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
new file mode 100644
index 0000000000..6c834be897
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+ install -d ${D}${bindir}
+ echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+ chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 315d1a61c2..9457f1e3ac 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -35,3 +35,13 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
TESTSTRING:pn-sysroot-test-arch2 = "%s"
""" % (uuid1, uuid2))
bitbake("sysroot-test")
+
+ def test_sysroot_max_shebang(self):
+ """
+ Summary: Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+ Expected: Fail when a shebang bigger than the max shebang-size is reached.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+ res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow
2022-06-07 15:11 ` [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow Paulo Neves
@ 2022-06-08 12:53 ` Luca Ceresoli
2022-06-08 14:45 ` Paulo Neves
[not found] ` <16F6A653C457F871.26104@lists.openembedded.org>
2022-06-15 11:04 ` Ross Burton
2 siblings, 1 reply; 64+ messages in thread
From: Luca Ceresoli @ 2022-06-08 12:53 UTC (permalink / raw)
To: Paulo Neves; +Cc: openembedded-core
Hi Paulo,
On Tue, 7 Jun 2022 17:11:22 +0200
"Paulo Neves" <ptsneves@gmail.com> wrote:
> Make sure we do not stage any executable with a bigger shebang
> than 128. Fixes [1]
>
> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
>
> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
This check seems to be working very well!! It triggered a huge amount
of build failures on the autobuilders due to libcheck having a shebang
too long in the checkmk script, e.g.:
#! /home/pokybuild/yocto-worker/genericx86-64-alt/build/build/tmp/work/x86_64-linux/libcheck-native/0.15.2-r0/recipe-sysroot-native/usr/bin/gawk -f
Here are a few logs:
https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/14/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/11/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/11/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/12/logs/errors
It would be great if you could add another patch to your series to fix
libcheck, and also to do 'bitbake world' to test as many packages as
possible before discovering from the autobuilders.
Thank you!
--
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow
2022-06-08 12:53 ` [OE-core] " Luca Ceresoli
@ 2022-06-08 14:45 ` Paulo Neves
2022-06-08 15:02 ` Luca Ceresoli
0 siblings, 1 reply; 64+ messages in thread
From: Paulo Neves @ 2022-06-08 14:45 UTC (permalink / raw)
To: Luca Ceresoli; +Cc: openembedded-core
On 6/8/22 14:53, Luca Ceresoli wrote:
> Hi Paulo,
>
> On Tue, 7 Jun 2022 17:11:22 +0200
> "Paulo Neves" <ptsneves@gmail.com> wrote:
>
>> Make sure we do not stage any executable with a bigger shebang
>> than 128. Fixes [1]
>>
>> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
>>
>> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> This check seems to be working very well!! It triggered a huge amount
> of build failures on the autobuilders due to libcheck having a shebang
> too long in the checkmk script, e.g.:
>
> #! /home/pokybuild/yocto-worker/genericx86-64-alt/build/build/tmp/work/x86_64-linux/libcheck-native/0.15.2-r0/recipe-sysroot-native/usr/bin/gawk -f
>
> Here are a few logs:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/14/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/11/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/11/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/12/logs/errors
I am not seeing an immediate way to fix this optimally. The go-to
solution to this class of issues is to just make
the #!/usr/bin/env <interpreter>. The issue is that there is an extra
-f, which with /usr/bin/env, will not work. The awk manual also implies
this is not optional for standalone scripts. I think we can create a
wrapper, or maybe we already have such a wrapper?
>
> It would be great if you could add another patch to your series to fix
> libcheck, and also to do 'bitbake world' to test as many packages as
> possible before discovering from the autobuilders.
It takes quite a while on my computer and often i get out of disk space.
I tried asking the linaro guys for a tuxsuite token but no answer yet.
If you have some way of getting resources to make builds let me know.
> Thank you!
Thank you
Paulo Neves
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow
2022-06-08 14:45 ` Paulo Neves
@ 2022-06-08 15:02 ` Luca Ceresoli
0 siblings, 0 replies; 64+ messages in thread
From: Luca Ceresoli @ 2022-06-08 15:02 UTC (permalink / raw)
To: Paulo Neves; +Cc: openembedded-core
On Wed, 8 Jun 2022 16:45:20 +0200
"Paulo Neves" <ptsneves@gmail.com> wrote:
> On 6/8/22 14:53, Luca Ceresoli wrote:
> > Hi Paulo,
> >
> > On Tue, 7 Jun 2022 17:11:22 +0200
> > "Paulo Neves" <ptsneves@gmail.com> wrote:
> >
> >> Make sure we do not stage any executable with a bigger shebang
> >> than 128. Fixes [1]
> >>
> >> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
> >>
> >> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> > This check seems to be working very well!! It triggered a huge amount
> > of build failures on the autobuilders due to libcheck having a shebang
> > too long in the checkmk script, e.g.:
> >
> > #! /home/pokybuild/yocto-worker/genericx86-64-alt/build/build/tmp/work/x86_64-linux/libcheck-native/0.15.2-r0/recipe-sysroot-native/usr/bin/gawk -f
> >
> > Here are a few logs:
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/14/logs/errors
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/11/logs/errors
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/11/logs/errors
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/12/logs/errors
>
> I am not seeing an immediate way to fix this optimally. The go-to
> solution to this class of issues is to just make
> the #!/usr/bin/env <interpreter>. The issue is that there is an extra
> -f, which with /usr/bin/env, will not work. The awk manual also implies
> this is not optional for standalone scripts. I think we can create a
> wrapper, or maybe we already have such a wrapper?
> >
> > It would be great if you could add another patch to your series to fix
> > libcheck, and also to do 'bitbake world' to test as many packages as
> > possible before discovering from the autobuilders.
> It takes quite a while on my computer and often i get out of disk space.
> I tried asking the linaro guys for a tuxsuite token but no answer yet.
> If you have some way of getting resources to make builds let me know.
Did you enable rm_work?
--
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 64+ messages in thread
[parent not found: <16F6A653C457F871.26104@lists.openembedded.org>]
* Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow
[not found] ` <16F6A653C457F871.26104@lists.openembedded.org>
@ 2022-06-08 15:06 ` Luca Ceresoli
2022-06-13 11:20 ` Paulo Neves
0 siblings, 1 reply; 64+ messages in thread
From: Luca Ceresoli @ 2022-06-08 15:06 UTC (permalink / raw)
To: Luca Ceresoli via lists.openembedded.org
Cc: luca.ceresoli, Paulo Neves, openembedded-core
Hi Paulo,
On Wed, 8 Jun 2022 14:53:05 +0200
"Luca Ceresoli via lists.openembedded.org"
<luca.ceresoli=bootlin.com@lists.openembedded.org> wrote:
> Hi Paulo,
>
> On Tue, 7 Jun 2022 17:11:22 +0200
> "Paulo Neves" <ptsneves@gmail.com> wrote:
>
> > Make sure we do not stage any executable with a bigger shebang
> > than 128. Fixes [1]
> >
> > [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
> >
> > Signed-off-by: Paulo Neves <ptsneves@gmail.com>
>
> This check seems to be working very well!! It triggered a huge amount
> of build failures on the autobuilders due to libcheck having a shebang
> too long in the checkmk script, e.g.:
>
> #! /home/pokybuild/yocto-worker/genericx86-64-alt/build/build/tmp/work/x86_64-linux/libcheck-native/0.15.2-r0/recipe-sysroot-native/usr/bin/gawk -f
>
> Here are a few logs:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/14/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/11/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/11/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/12/logs/errors
>
> It would be great if you could add another patch to your series to fix
> libcheck, and also to do 'bitbake world' to test as many packages as
> possible before discovering from the autobuilders.
Here are more failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/12/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/5340/steps/12/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/5340/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/13/logs/errors
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/13/logs/stdio
This time the error is:
stdio: ERROR: core-image-sato-1.0-r0 do_testsdk: The toolchain <...> is not built. Build it before running the tests: 'bitbake <image> -c populate_sdk' .
I'm not sure exactly how your code triggers such error, but it appeared
when testing on the autobuilders with this patch series and disappeared
when I removed only these 5 patches, thus it seems related.
--
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow
2022-06-08 15:06 ` Luca Ceresoli
@ 2022-06-13 11:20 ` Paulo Neves
0 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-13 11:20 UTC (permalink / raw)
To: Luca Ceresoli, Luca Ceresoli via lists.openembedded.org; +Cc: openembedded-core
Hey Luca,
I sent a new patchset and from my tests it all should be good now.
Paulo Neves
On 6/8/22 17:06, Luca Ceresoli wrote:
> Hi Paulo,
>
> On Wed, 8 Jun 2022 14:53:05 +0200
> "Luca Ceresoli via lists.openembedded.org"
> <luca.ceresoli=bootlin.com@lists.openembedded.org> wrote:
>
>> Hi Paulo,
>>
>> On Tue, 7 Jun 2022 17:11:22 +0200
>> "Paulo Neves" <ptsneves@gmail.com> wrote:
>>
>>> Make sure we do not stage any executable with a bigger shebang
>>> than 128. Fixes [1]
>>>
>>> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
>>>
>>> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
>> This check seems to be working very well!! It triggered a huge amount
>> of build failures on the autobuilders due to libcheck having a shebang
>> too long in the checkmk script, e.g.:
>>
>> #! /home/pokybuild/yocto-worker/genericx86-64-alt/build/build/tmp/work/x86_64-linux/libcheck-native/0.15.2-r0/recipe-sysroot-native/usr/bin/gawk -f
>>
>> Here are a few logs:
>>
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/14/logs/errors
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/5367/steps/11/logs/errors
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/11/logs/errors
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/12/logs/errors
>>
>> It would be great if you could add another patch to your series to fix
>> libcheck, and also to do 'bitbake world' to test as many packages as
>> possible before discovering from the autobuilders.
> Here are more failures:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/12/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/5689/steps/12/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/5340/steps/12/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/62/builds/5340/steps/12/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/13/logs/errors
> https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5340/steps/13/logs/stdio
>
> This time the error is:
>
> stdio: ERROR: core-image-sato-1.0-r0 do_testsdk: The toolchain <...> is not built. Build it before running the tests: 'bitbake <image> -c populate_sdk' .
>
> I'm not sure exactly how your code triggers such error, but it appeared
> when testing on the autobuilders with this patch series and disappeared
> when I removed only these 5 patches, thus it seems related.
>
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: [OE-core] [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow
2022-06-07 15:11 ` [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow Paulo Neves
2022-06-08 12:53 ` [OE-core] " Luca Ceresoli
[not found] ` <16F6A653C457F871.26104@lists.openembedded.org>
@ 2022-06-15 11:04 ` Ross Burton
2 siblings, 0 replies; 64+ messages in thread
From: Ross Burton @ 2022-06-15 11:04 UTC (permalink / raw)
To: ptsneves@gmail.com; +Cc: openembedded-core@lists.openembedded.org
> + expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
> + res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
> + self.assertTrue(expected in res.output, msg=res.output)
I err towards paranoia and would also check that res.status != 0.
Ross
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
^ permalink raw reply [flat|nested] 64+ messages in thread
* [PATCH v2 5/5] oeqa/selftest: Test staged .la and .pc files
2022-06-07 15:11 ` [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py Paulo Neves
` (2 preceding siblings ...)
2022-06-07 15:11 ` [PATCH v2 4/5] oeqa/selftest: Add test for shebang overflow Paulo Neves
@ 2022-06-07 15:11 ` Paulo Neves
2022-06-07 16:53 ` [OE-core] [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py Alexander Kanavin
4 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-07 15:11 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
These files are checked by qa_check_staged but there was no
test cases for whether the tests actually worked. Now there
are.
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
.../sysroot-test/sysroot-la-test_1.0.bb | 16 ++++++++++
.../sysroot-test/sysroot-pc-test_1.0.bb | 12 +++++++
meta/lib/oeqa/selftest/cases/sysroot.py | 32 +++++++++++++++++++
3 files changed, 60 insertions(+)
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
new file mode 100644
index 0000000000..21f06782fb
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+ install -d ${D}${libdir}/test/
+ echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
new file mode 100644
index 0000000000..e748310fc4
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+ install -d ${D}${libdir}/test/
+ echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 9457f1e3ac..588fc8c713 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -45,3 +45,35 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
self.assertTrue(expected in res.output, msg=res.output)
+
+ def test_sysroot_la(self):
+ """
+ Summary: Check that workdir paths are not contained in .la files.
+ Expected: Fail when a workdir path is found in the file content.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "la-test.la failed sanity test (workdir) in path"
+
+ res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[la]' in res.output, msg=res.output)
+
+ res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[la]' in res.output, msg=res.output)
+
+ def test_sysroot_pkgconfig(self):
+ """
+ Summary: Check that tmpdir paths are not contained in .pc files.
+ Expected: Fail when a tmpdir path is found in the file content.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "test.pc failed sanity test (tmpdir) in path"
+
+ res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+ self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+ self.assertTrue(expected in res.output, msg=res.output)
+
+ res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py
2022-06-07 15:11 ` [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py Paulo Neves
` (3 preceding siblings ...)
2022-06-07 15:11 ` [PATCH v2 5/5] oeqa/selftest: Test staged .la and .pc files Paulo Neves
@ 2022-06-07 16:53 ` Alexander Kanavin
2022-06-07 20:03 ` Paulo Neves
4 siblings, 1 reply; 64+ messages in thread
From: Alexander Kanavin @ 2022-06-07 16:53 UTC (permalink / raw)
To: Paulo Neves; +Cc: OE-core
I'm not sure this is Inappropriate. Can you send this to upstream nevertheless?
Alex
On Tue, 7 Jun 2022 at 17:11, Paulo Neves <ptsneves@gmail.com> wrote:
>
> The native path may be too big, leading to shebang
> overflow. Just use the #!/usr/bin/env python3.
>
> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> ---
> ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
> .../recipes-devtools/python/python3_3.10.4.bb | 1 +
> 2 files changed, 30 insertions(+)
> create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>
> diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> new file mode 100644
> index 0000000000..9f23278a30
> --- /dev/null
> +++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> @@ -0,0 +1,29 @@
> +From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
> +From: Paulo Neves <ptsneves@gmail.com>
> +Date: Tue, 7 Jun 2022 16:16:41 +0200
> +Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
> +
> +The whole native path may be too big, leading to shebang
> +overflow. Let's just use the env shebang.
> +
> +Upstream-Status: Inappropriate [distribution]
> +---
> + Makefile.pre.in | 2 ++
> + 1 file changed, 2 insertions(+)
> +
> +diff --git a/Makefile.pre.in b/Makefile.pre.in
> +index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
> +--- a/Makefile.pre.in
> ++++ b/Makefile.pre.in
> +@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
> + @ # Substitution happens here, as the completely-expanded BINDIR
> + @ # is not available in configure
> + sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
> ++ @ # Otherwise we might get huge shebangs with native paths
> ++ sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
> + @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
> + LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
> + @ # In OpenEmbedded, always use the python version of the script, the shell
> +--
> +2.25.1
> +
> diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
> index 6bd3a6aba8..357025f856 100644
> --- a/meta/recipes-devtools/python/python3_3.10.4.bb
> +++ b/meta/recipes-devtools/python/python3_3.10.4.bb
> @@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
> file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
> file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
> file://deterministic_imports.patch \
> + file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
> "
>
> SRC_URI:append:class-native = " \
> --
> 2.25.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#166670): https://lists.openembedded.org/g/openembedded-core/message/166670
> Mute This Topic: https://lists.openembedded.org/mt/91602889/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
^ permalink raw reply [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py
2022-06-07 16:53 ` [OE-core] [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py Alexander Kanavin
@ 2022-06-07 20:03 ` Paulo Neves
2022-06-07 22:09 ` Alexander Kanavin
0 siblings, 1 reply; 64+ messages in thread
From: Paulo Neves @ 2022-06-07 20:03 UTC (permalink / raw)
To: Alexander Kanavin; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 4038 bytes --]
I think this patch will never be accepted because it hard-codes and depends
on env. I recall that red hat for example does not accept the use of env,
possibly for security reasons.
Paulo Neves
On Tue, Jun 7, 2022, 18:53 Alexander Kanavin <alex.kanavin@gmail.com> wrote:
> I'm not sure this is Inappropriate. Can you send this to upstream
> nevertheless?
>
> Alex
>
> On Tue, 7 Jun 2022 at 17:11, Paulo Neves <ptsneves@gmail.com> wrote:
> >
> > The native path may be too big, leading to shebang
> > overflow. Just use the #!/usr/bin/env python3.
> >
> > Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> > ---
> > ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
> > .../recipes-devtools/python/python3_3.10.4.bb | 1 +
> > 2 files changed, 30 insertions(+)
> > create mode 100644
> meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> >
> > diff --git
> a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> > new file mode 100644
> > index 0000000000..9f23278a30
> > --- /dev/null
> > +++
> b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> > @@ -0,0 +1,29 @@
> > +From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
> > +From: Paulo Neves <ptsneves@gmail.com>
> > +Date: Tue, 7 Jun 2022 16:16:41 +0200
> > +Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
> > +
> > +The whole native path may be too big, leading to shebang
> > +overflow. Let's just use the env shebang.
> > +
> > +Upstream-Status: Inappropriate [distribution]
> > +---
> > + Makefile.pre.in | 2 ++
> > + 1 file changed, 2 insertions(+)
> > +
> > +diff --git a/Makefile.pre.in b/Makefile.pre.in
> > +index
> f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176
> 100644
> > +--- a/Makefile.pre.in
> > ++++ b/Makefile.pre.in
> > +@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in
> Misc/python-config.sh
> > + @ # Substitution happens here, as the completely-expanded BINDIR
> > + @ # is not available in configure
> > + sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," <
> $(srcdir)/Misc/python-config.in >python-config.py
> > ++ @ # Otherwise we might get huge shebangs with native paths
> > ++ sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
> > + @ # Replace makefile compat. variable references with shell
> script compat. ones; $(VAR) -> ${VAR}
> > + LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' <
> Misc/python-config.sh >python-config
> > + @ # In OpenEmbedded, always use the python version of the
> script, the shell
> > +--
> > +2.25.1
> > +
> > diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb
> b/meta/recipes-devtools/python/python3_3.10.4.bb
> > index 6bd3a6aba8..357025f856 100644
> > --- a/meta/recipes-devtools/python/python3_3.10.4.bb
> > +++ b/meta/recipes-devtools/python/python3_3.10.4.bb
> > @@ -34,6 +34,7 @@ SRC_URI = "
> http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
> >
> file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
> >
> file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
> > file://deterministic_imports.patch \
> > + file://0001-Avoid-shebang-overflow-on-python-config.py.patch
> \
> > "
> >
> > SRC_URI:append:class-native = " \
> > --
> > 2.25.1
> >
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#166670):
> https://lists.openembedded.org/g/openembedded-core/message/166670
> > Mute This Topic: https://lists.openembedded.org/mt/91602889/1686489
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
[-- Attachment #2: Type: text/html, Size: 6776 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py
2022-06-07 20:03 ` Paulo Neves
@ 2022-06-07 22:09 ` Alexander Kanavin
0 siblings, 0 replies; 64+ messages in thread
From: Alexander Kanavin @ 2022-06-07 22:09 UTC (permalink / raw)
To: Paulo Neves; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 4477 bytes --]
Please submit it nevertheless. If upstream says no, we'll just mark it as
denied, but it’s not appropriate to declare it inappropriate a priori.
Alex
On Tue 7. Jun 2022 at 23.03, Paulo Neves <ptsneves@gmail.com> wrote:
> I think this patch will never be accepted because it hard-codes and
> depends on env. I recall that red hat for example does not accept the use
> of env, possibly for security reasons.
>
> Paulo Neves
>
> On Tue, Jun 7, 2022, 18:53 Alexander Kanavin <alex.kanavin@gmail.com>
> wrote:
>
>> I'm not sure this is Inappropriate. Can you send this to upstream
>> nevertheless?
>>
>> Alex
>>
>> On Tue, 7 Jun 2022 at 17:11, Paulo Neves <ptsneves@gmail.com> wrote:
>> >
>> > The native path may be too big, leading to shebang
>> > overflow. Just use the #!/usr/bin/env python3.
>> >
>> > Signed-off-by: Paulo Neves <ptsneves@gmail.com>
>> > ---
>> > ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
>> > .../recipes-devtools/python/python3_3.10.4.bb | 1 +
>> > 2 files changed, 30 insertions(+)
>> > create mode 100644
>> meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>> >
>> > diff --git
>> a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>> b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>> > new file mode 100644
>> > index 0000000000..9f23278a30
>> > --- /dev/null
>> > +++
>> b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>> > @@ -0,0 +1,29 @@
>> > +From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
>> > +From: Paulo Neves <ptsneves@gmail.com>
>> > +Date: Tue, 7 Jun 2022 16:16:41 +0200
>> > +Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
>> > +
>> > +The whole native path may be too big, leading to shebang
>> > +overflow. Let's just use the env shebang.
>> > +
>> > +Upstream-Status: Inappropriate [distribution]
>> > +---
>> > + Makefile.pre.in | 2 ++
>> > + 1 file changed, 2 insertions(+)
>> > +
>> > +diff --git a/Makefile.pre.in b/Makefile.pre.in
>> > +index
>> f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176
>> 100644
>> > +--- a/Makefile.pre.in
>> > ++++ b/Makefile.pre.in
>> > +@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in
>> Misc/python-config.sh
>> > + @ # Substitution happens here, as the completely-expanded BINDIR
>> > + @ # is not available in configure
>> > + sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," <
>> $(srcdir)/Misc/python-config.in >python-config.py
>> > ++ @ # Otherwise we might get huge shebangs with native paths
>> > ++ sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
>> > + @ # Replace makefile compat. variable references with shell
>> script compat. ones; $(VAR) -> ${VAR}
>> > + LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' <
>> Misc/python-config.sh >python-config
>> > + @ # In OpenEmbedded, always use the python version of the
>> script, the shell
>> > +--
>> > +2.25.1
>> > +
>> > diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb
>> b/meta/recipes-devtools/python/python3_3.10.4.bb
>> > index 6bd3a6aba8..357025f856 100644
>> > --- a/meta/recipes-devtools/python/python3_3.10.4.bb
>> > +++ b/meta/recipes-devtools/python/python3_3.10.4.bb
>> > @@ -34,6 +34,7 @@ SRC_URI = "
>> http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
>> >
>> file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
>> >
>> file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
>> > file://deterministic_imports.patch \
>> > +
>> file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
>> > "
>> >
>> > SRC_URI:append:class-native = " \
>> > --
>> > 2.25.1
>> >
>> >
>> > -=-=-=-=-=-=-=-=-=-=-=-
>> > Links: You receive all messages sent to this group.
>> > View/Reply Online (#166670):
>> https://lists.openembedded.org/g/openembedded-core/message/166670
>> > Mute This Topic: https://lists.openembedded.org/mt/91602889/1686489
>> > Group Owner: openembedded-core+owner@lists.openembedded.org
>> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
>> alex.kanavin@gmail.com]
>> > -=-=-=-=-=-=-=-=-=-=-=-
>> >
>>
>
[-- Attachment #2: Type: text/html, Size: 7481 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* [PATCH v3 1/7] python: Avoid shebang overflow on python-config.py
2022-06-06 15:58 [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
` (2 preceding siblings ...)
2022-06-07 15:11 ` [PATCH v2 1/5] python: Avoid shebang overflow on python-config.py Paulo Neves
@ 2022-06-10 21:43 ` Paulo Neves
2022-06-10 21:43 ` [PATCH v3 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2 Paulo Neves
` (6 more replies)
2022-06-14 13:16 ` [PATCH v4 " Paulo Neves
` (3 subsequent siblings)
7 siblings, 7 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-10 21:43 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
The native path may be too big, leading to shebang
overflow. Just use the #!/usr/bin/env python3.
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
.../recipes-devtools/python/python3_3.10.4.bb | 1 +
2 files changed, 30 insertions(+)
create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
new file mode 100644
index 0000000000..9f23278a30
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
@@ -0,0 +1,29 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <ptsneves@gmail.com>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Upstream-Status: Inappropriate [distribution]
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+ @ # Substitution happens here, as the completely-expanded BINDIR
+ @ # is not available in configure
+ sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++ @ # Otherwise we might get huge shebangs with native paths
++ sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+ @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+ LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+ @ # In OpenEmbedded, always use the python version of the script, the shell
+--
+2.25.1
+
diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
index 6bd3a6aba8..357025f856 100644
--- a/meta/recipes-devtools/python/python3_3.10.4.bb
+++ b/meta/recipes-devtools/python/python3_3.10.4.bb
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
file://deterministic_imports.patch \
+ file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
"
SRC_URI:append:class-native = " \
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v3 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2
2022-06-10 21:43 ` [PATCH v3 1/7] " Paulo Neves
@ 2022-06-10 21:43 ` Paulo Neves
2022-06-10 21:43 ` [PATCH v3 3/7] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
` (5 subsequent siblings)
6 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-10 21:43 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
A native path can cause a shebang overflow on gtkdoc-mkhtml.
Replace it with /usr/bin/env.
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
index 150d2c0b23..392913fcc6 100644
--- a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
+++ b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
@@ -43,6 +43,7 @@ do_install:append () {
${datadir}/gtk-doc/python/gtkdoc/config.py; do
sed -e 's,${RECIPE_SYSROOT_NATIVE}/usr/bin/pkg-config,${bindir}/pkg-config,' \
-e 's,${HOSTTOOLS_DIR}/python3,${bindir}/python3,' \
+ -e '1s|^#!.*|#!/usr/bin/env python3|' \
-i ${D}$fn
done
}
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v3 3/7] insane.bbclass: Make do_qa_staging check shebangs
2022-06-10 21:43 ` [PATCH v3 1/7] " Paulo Neves
2022-06-10 21:43 ` [PATCH v3 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2 Paulo Neves
@ 2022-06-10 21:43 ` Paulo Neves
2022-06-10 21:43 ` [PATCH v3 4/7] oeqa/selftest: Add test for shebang overflow Paulo Neves
` (4 subsequent siblings)
6 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-10 21:43 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
As reported in the bug report [1], there was no check for shebang
sizes on native scripts and now this is fixed.
The path scope of the qa_staging was increased from just checking
libdir to all the relevant SYSROOT_DIRS.
It is possible to skip this check through INSANE_SKIP.
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
meta/classes/insane.bbclass | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 9ca84bace9..b2951a48fe 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
skip_pkgconfig = True
+ skip_shebang_size = False
+ if 'shebang-size' in skip:
+ bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+ skip_shebang_size = True
+
# find all .la and .pc files
# read the content
# and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
oe.qa.handle_error("pkgconfig", error_msg, d)
+ if not skip_shebang_size:
+ errors = {}
+ package_qa_check_shebang_size(path, "", d, None, errors)
+ for e in errors:
+ oe.qa.handle_error(e, errors[e], d)
+
+
# Run all package-wide warnfuncs and errorfuncs
def package_qa_package(warnfuncs, errorfuncs, package, d):
warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
python do_qa_staging() {
bb.note("QA checking staging")
- qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+ sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+ for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+ qa_check_staged(sysroot_destdir + sysroot_dir, d)
oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
}
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v3 4/7] oeqa/selftest: Add test for shebang overflow
2022-06-10 21:43 ` [PATCH v3 1/7] " Paulo Neves
2022-06-10 21:43 ` [PATCH v3 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2 Paulo Neves
2022-06-10 21:43 ` [PATCH v3 3/7] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
@ 2022-06-10 21:43 ` Paulo Neves
2022-06-10 21:43 ` [PATCH v3 5/7] oeqa/selftest: Test staged .la and .pc files Paulo Neves
` (3 subsequent siblings)
6 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-10 21:43 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
Make sure we do not stage any executable with a bigger shebang
than 128. Fixes [1]
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
.../sysroot-test/sysroot-shebang-test_1.0.bb | 12 ++++++++++++
meta/lib/oeqa/selftest/cases/sysroot.py | 10 ++++++++++
2 files changed, 22 insertions(+)
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
new file mode 100644
index 0000000000..6c834be897
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+ install -d ${D}${bindir}
+ echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+ chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 315d1a61c2..9457f1e3ac 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -35,3 +35,13 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
TESTSTRING:pn-sysroot-test-arch2 = "%s"
""" % (uuid1, uuid2))
bitbake("sysroot-test")
+
+ def test_sysroot_max_shebang(self):
+ """
+ Summary: Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+ Expected: Fail when a shebang bigger than the max shebang-size is reached.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+ res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v3 5/7] oeqa/selftest: Test staged .la and .pc files
2022-06-10 21:43 ` [PATCH v3 1/7] " Paulo Neves
` (2 preceding siblings ...)
2022-06-10 21:43 ` [PATCH v3 4/7] oeqa/selftest: Add test for shebang overflow Paulo Neves
@ 2022-06-10 21:43 ` Paulo Neves
2022-06-10 21:43 ` [PATCH v3 6/7] utils: Add cmdline_shebang_wrapper util Paulo Neves
` (2 subsequent siblings)
6 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-10 21:43 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
These files are checked by qa_check_staged but there was no
test cases for whether the tests actually worked. Now there
are.
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
.../sysroot-test/sysroot-la-test_1.0.bb | 16 ++++++++++
.../sysroot-test/sysroot-pc-test_1.0.bb | 12 +++++++
meta/lib/oeqa/selftest/cases/sysroot.py | 32 +++++++++++++++++++
3 files changed, 60 insertions(+)
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
new file mode 100644
index 0000000000..21f06782fb
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+ install -d ${D}${libdir}/test/
+ echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
new file mode 100644
index 0000000000..e748310fc4
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+ install -d ${D}${libdir}/test/
+ echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 9457f1e3ac..588fc8c713 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -45,3 +45,35 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
self.assertTrue(expected in res.output, msg=res.output)
+
+ def test_sysroot_la(self):
+ """
+ Summary: Check that workdir paths are not contained in .la files.
+ Expected: Fail when a workdir path is found in the file content.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "la-test.la failed sanity test (workdir) in path"
+
+ res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[la]' in res.output, msg=res.output)
+
+ res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[la]' in res.output, msg=res.output)
+
+ def test_sysroot_pkgconfig(self):
+ """
+ Summary: Check that tmpdir paths are not contained in .pc files.
+ Expected: Fail when a tmpdir path is found in the file content.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "test.pc failed sanity test (tmpdir) in path"
+
+ res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+ self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+ self.assertTrue(expected in res.output, msg=res.output)
+
+ res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v3 6/7] utils: Add cmdline_shebang_wrapper util.
2022-06-10 21:43 ` [PATCH v3 1/7] " Paulo Neves
` (3 preceding siblings ...)
2022-06-10 21:43 ` [PATCH v3 5/7] oeqa/selftest: Test staged .la and .pc files Paulo Neves
@ 2022-06-10 21:43 ` Paulo Neves
2022-06-10 21:43 ` [PATCH v3 7/7] libcheck: Fix too long shebang for native case Paulo Neves
2022-06-13 7:30 ` [OE-core] [PATCH v3 1/7] python: Avoid shebang overflow on python-config.py Alexander Kanavin
6 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-10 21:43 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
Useful to work around shebang relocation issues, where
shebangs are too long or have arguments in them, thus preventing them
from using the /usr/bin/env shebang.
---
.../wrapper/cmdline-shebang-wrapper-test.bb | 21 ++++++++++++
.../recipes-test/wrapper/files/test.awk | 2 ++
meta/classes/utils.bbclass | 34 +++++++++++++++++++
meta/lib/oeqa/selftest/cases/wrapper.py | 11 ++++++
4 files changed, 68 insertions(+)
create mode 100644 meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
diff --git a/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
new file mode 100644
index 0000000000..302eea8901
--- /dev/null
+++ b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
@@ -0,0 +1,21 @@
+FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
+SUMMARY = "Check that create_cmdline_shebang works"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI += "file://test.awk"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+ install -d ${D}${bindir}
+ install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
+ sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
+ create_cmdline_shebang_wrapper ${D}${bindir}/test
+ if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
+ bbfatal "Wrapper is broken"
+ else
+ bbnote "Wrapper is good"
+ fi
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk b/meta-selftest/recipes-test/wrapper/files/test.awk
new file mode 100644
index 0000000000..91429197b1
--- /dev/null
+++ b/meta-selftest/recipes-test/wrapper/files/test.awk
@@ -0,0 +1,2 @@
+#! @AWK_BIN@ -f
+BEGIN { print "Don't Panic!" }
diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
index b4eb3d38ab..b617632d9f 100644
--- a/meta/classes/utils.bbclass
+++ b/meta/classes/utils.bbclass
@@ -184,6 +184,40 @@ END
chmod +x $cmd
}
+create_cmdline_shebang_wrapper () {
+ # Create a wrapper script where commandline options are needed
+ #
+ # These are useful to work around shebang relocation issues, where shebangs are too
+ # long or have arguments in them, thus preventing them from using the /usr/bin/env
+ # shebang
+ #
+ # Usage: create_cmdline_wrapper FILENAME <extra-options>
+
+ cmd=$1
+ shift
+
+ echo "Generating wrapper script for $cmd"
+
+ # Strip #! and get remaining interpreter + arg
+ argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
+ # strip the shebang from the real script as we do not want it to be usable anyway
+ tail -n +2 $cmd > $cmd.real
+ cmdname=$(basename $cmd)
+ dirname=$(dirname $cmd)
+ cmdoptions=$@
+ if [ "${base_prefix}" != "" ]; then
+ relpath=`python3 -c "import os; print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
+ cmdoptions=`echo $@ | sed -e "s:${base_prefix}:\\$realdir/$relpath:g"`
+ fi
+ cat <<END >$cmd
+#!/usr/bin/env bash
+realpath=\`readlink -fn \$0\`
+realdir=\`dirname \$realpath\`
+exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions "\$@"
+END
+ chmod +x $cmd
+}
+
create_wrapper () {
# Create a wrapper script where extra environment variables are needed
#
diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py b/meta/lib/oeqa/selftest/cases/wrapper.py
new file mode 100644
index 0000000000..6de63310c0
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/wrapper.py
@@ -0,0 +1,11 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake
+
+class WrapperTests(OESelftestTestCase):
+ def test_shebang_wrapper(self):
+ """
+ Summary: Build a recipe which will fail if the cmdline_shebang_wrapper function is defective.
+ Expected: Exit status to be 0.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ res = bitbake("cmdline-shebang-wrapper-test -c install", ignore_status=False)
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v3 7/7] libcheck: Fix too long shebang for native case.
2022-06-10 21:43 ` [PATCH v3 1/7] " Paulo Neves
` (4 preceding siblings ...)
2022-06-10 21:43 ` [PATCH v3 6/7] utils: Add cmdline_shebang_wrapper util Paulo Neves
@ 2022-06-10 21:43 ` Paulo Neves
2022-06-13 7:30 ` [OE-core] [PATCH v3 1/7] python: Avoid shebang overflow on python-config.py Alexander Kanavin
6 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-10 21:43 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
It requires a shebang wrapper due to the fact that awk interpreter
has an argument.
---
meta/recipes-support/libcheck/libcheck_0.15.2.bb | 3 +++
1 file changed, 3 insertions(+)
diff --git a/meta/recipes-support/libcheck/libcheck_0.15.2.bb b/meta/recipes-support/libcheck/libcheck_0.15.2.bb
index 188d689cc3..1393aa2a1c 100644
--- a/meta/recipes-support/libcheck/libcheck_0.15.2.bb
+++ b/meta/recipes-support/libcheck/libcheck_0.15.2.bb
@@ -23,6 +23,9 @@ CACHED_CONFIGUREVARS += "ac_cv_path_AWK_PATH=${bindir}/gawk"
RREPLACES:${PN} = "check (<= 0.9.5)"
+do_install:append:class-native() {
+ create_cmdline_shebang_wrapper ${D}${bindir}/checkmk
+}
BBCLASSEXTEND = "native nativesdk"
PACKAGES =+ "checkmk"
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v3 1/7] python: Avoid shebang overflow on python-config.py
2022-06-10 21:43 ` [PATCH v3 1/7] " Paulo Neves
` (5 preceding siblings ...)
2022-06-10 21:43 ` [PATCH v3 7/7] libcheck: Fix too long shebang for native case Paulo Neves
@ 2022-06-13 7:30 ` Alexander Kanavin
2022-06-13 10:32 ` Paulo Neves
6 siblings, 1 reply; 64+ messages in thread
From: Alexander Kanavin @ 2022-06-13 7:30 UTC (permalink / raw)
To: Paulo Neves; +Cc: OE-core
As requested, please submit upstream.
Alex
On Fri, 10 Jun 2022 at 23:43, Paulo Neves <ptsneves@gmail.com> wrote:
>
> The native path may be too big, leading to shebang
> overflow. Just use the #!/usr/bin/env python3.
>
> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> ---
> ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
> .../recipes-devtools/python/python3_3.10.4.bb | 1 +
> 2 files changed, 30 insertions(+)
> create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>
> diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> new file mode 100644
> index 0000000000..9f23278a30
> --- /dev/null
> +++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> @@ -0,0 +1,29 @@
> +From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
> +From: Paulo Neves <ptsneves@gmail.com>
> +Date: Tue, 7 Jun 2022 16:16:41 +0200
> +Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
> +
> +The whole native path may be too big, leading to shebang
> +overflow. Let's just use the env shebang.
> +
> +Upstream-Status: Inappropriate [distribution]
> +---
> + Makefile.pre.in | 2 ++
> + 1 file changed, 2 insertions(+)
> +
> +diff --git a/Makefile.pre.in b/Makefile.pre.in
> +index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
> +--- a/Makefile.pre.in
> ++++ b/Makefile.pre.in
> +@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
> + @ # Substitution happens here, as the completely-expanded BINDIR
> + @ # is not available in configure
> + sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
> ++ @ # Otherwise we might get huge shebangs with native paths
> ++ sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
> + @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
> + LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
> + @ # In OpenEmbedded, always use the python version of the script, the shell
> +--
> +2.25.1
> +
> diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
> index 6bd3a6aba8..357025f856 100644
> --- a/meta/recipes-devtools/python/python3_3.10.4.bb
> +++ b/meta/recipes-devtools/python/python3_3.10.4.bb
> @@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
> file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
> file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
> file://deterministic_imports.patch \
> + file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
> "
>
> SRC_URI:append:class-native = " \
> --
> 2.25.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#166813): https://lists.openembedded.org/g/openembedded-core/message/166813
> Mute This Topic: https://lists.openembedded.org/mt/91678611/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
^ permalink raw reply [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v3 1/7] python: Avoid shebang overflow on python-config.py
2022-06-13 7:30 ` [OE-core] [PATCH v3 1/7] python: Avoid shebang overflow on python-config.py Alexander Kanavin
@ 2022-06-13 10:32 ` Paulo Neves
2022-06-13 12:36 ` Alexander Kanavin
0 siblings, 1 reply; 64+ messages in thread
From: Paulo Neves @ 2022-06-13 10:32 UTC (permalink / raw)
To: Alexander Kanavin; +Cc: OE-core
As requested
https://github.com/python/cpython/pull/93760
Paulo Neves
On 6/13/22 09:30, Alexander Kanavin wrote:
> As requested, please submit upstream.
>
> Alex
>
> On Fri, 10 Jun 2022 at 23:43, Paulo Neves <ptsneves@gmail.com> wrote:
>> The native path may be too big, leading to shebang
>> overflow. Just use the #!/usr/bin/env python3.
>>
>> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
>> ---
>> ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
>> .../recipes-devtools/python/python3_3.10.4.bb | 1 +
>> 2 files changed, 30 insertions(+)
>> create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>>
>> diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>> new file mode 100644
>> index 0000000000..9f23278a30
>> --- /dev/null
>> +++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
>> @@ -0,0 +1,29 @@
>> +From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
>> +From: Paulo Neves <ptsneves@gmail.com>
>> +Date: Tue, 7 Jun 2022 16:16:41 +0200
>> +Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
>> +
>> +The whole native path may be too big, leading to shebang
>> +overflow. Let's just use the env shebang.
>> +
>> +Upstream-Status: Inappropriate [distribution]
>> +---
>> + Makefile.pre.in | 2 ++
>> + 1 file changed, 2 insertions(+)
>> +
>> +diff --git a/Makefile.pre.in b/Makefile.pre.in
>> +index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
>> +--- a/Makefile.pre.in
>> ++++ b/Makefile.pre.in
>> +@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
>> + @ # Substitution happens here, as the completely-expanded BINDIR
>> + @ # is not available in configure
>> + sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
>> ++ @ # Otherwise we might get huge shebangs with native paths
>> ++ sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
>> + @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
>> + LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
>> + @ # In OpenEmbedded, always use the python version of the script, the shell
>> +--
>> +2.25.1
>> +
>> diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
>> index 6bd3a6aba8..357025f856 100644
>> --- a/meta/recipes-devtools/python/python3_3.10.4.bb
>> +++ b/meta/recipes-devtools/python/python3_3.10.4.bb
>> @@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
>> file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
>> file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
>> file://deterministic_imports.patch \
>> + file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
>> "
>>
>> SRC_URI:append:class-native = " \
>> --
>> 2.25.1
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#166813): https://lists.openembedded.org/g/openembedded-core/message/166813
>> Mute This Topic: https://lists.openembedded.org/mt/91678611/1686489
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
^ permalink raw reply [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v3 1/7] python: Avoid shebang overflow on python-config.py
2022-06-13 10:32 ` Paulo Neves
@ 2022-06-13 12:36 ` Alexander Kanavin
0 siblings, 0 replies; 64+ messages in thread
From: Alexander Kanavin @ 2022-06-13 12:36 UTC (permalink / raw)
To: Paulo Neves; +Cc: OE-core
Thanks, can you resend with updated Upstream-Status that includes the link?
Alex
On Mon, 13 Jun 2022 at 12:32, Paulo Neves <ptsneves@gmail.com> wrote:
>
> As requested
> https://github.com/python/cpython/pull/93760
>
> Paulo Neves
>
> On 6/13/22 09:30, Alexander Kanavin wrote:
> > As requested, please submit upstream.
> >
> > Alex
> >
> > On Fri, 10 Jun 2022 at 23:43, Paulo Neves <ptsneves@gmail.com> wrote:
> >> The native path may be too big, leading to shebang
> >> overflow. Just use the #!/usr/bin/env python3.
> >>
> >> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> >> ---
> >> ...shebang-overflow-on-python-config.py.patch | 29 +++++++++++++++++++
> >> .../recipes-devtools/python/python3_3.10.4.bb | 1 +
> >> 2 files changed, 30 insertions(+)
> >> create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> >>
> >> diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> >> new file mode 100644
> >> index 0000000000..9f23278a30
> >> --- /dev/null
> >> +++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
> >> @@ -0,0 +1,29 @@
> >> +From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
> >> +From: Paulo Neves <ptsneves@gmail.com>
> >> +Date: Tue, 7 Jun 2022 16:16:41 +0200
> >> +Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
> >> +
> >> +The whole native path may be too big, leading to shebang
> >> +overflow. Let's just use the env shebang.
> >> +
> >> +Upstream-Status: Inappropriate [distribution]
> >> +---
> >> + Makefile.pre.in | 2 ++
> >> + 1 file changed, 2 insertions(+)
> >> +
> >> +diff --git a/Makefile.pre.in b/Makefile.pre.in
> >> +index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
> >> +--- a/Makefile.pre.in
> >> ++++ b/Makefile.pre.in
> >> +@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
> >> + @ # Substitution happens here, as the completely-expanded BINDIR
> >> + @ # is not available in configure
> >> + sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
> >> ++ @ # Otherwise we might get huge shebangs with native paths
> >> ++ sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
> >> + @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
> >> + LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
> >> + @ # In OpenEmbedded, always use the python version of the script, the shell
> >> +--
> >> +2.25.1
> >> +
> >> diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
> >> index 6bd3a6aba8..357025f856 100644
> >> --- a/meta/recipes-devtools/python/python3_3.10.4.bb
> >> +++ b/meta/recipes-devtools/python/python3_3.10.4.bb
> >> @@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
> >> file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
> >> file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
> >> file://deterministic_imports.patch \
> >> + file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
> >> "
> >>
> >> SRC_URI:append:class-native = " \
> >> --
> >> 2.25.1
> >>
> >>
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >> Links: You receive all messages sent to this group.
> >> View/Reply Online (#166813): https://lists.openembedded.org/g/openembedded-core/message/166813
> >> Mute This Topic: https://lists.openembedded.org/mt/91678611/1686489
> >> Group Owner: openembedded-core+owner@lists.openembedded.org
> >> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> >> -=-=-=-=-=-=-=-=-=-=-=-
> >>
>
^ permalink raw reply [flat|nested] 64+ messages in thread
* [PATCH v4 1/7] python: Avoid shebang overflow on python-config.py
2022-06-06 15:58 [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
` (3 preceding siblings ...)
2022-06-10 21:43 ` [PATCH v3 1/7] " Paulo Neves
@ 2022-06-14 13:16 ` Paulo Neves
2022-06-14 13:16 ` [PATCH v4 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2 Paulo Neves
` (5 more replies)
2022-06-14 15:10 ` [PATCH v5 1/7] python: Avoid shebang overflow on python-config.py Paulo Neves
` (2 subsequent siblings)
7 siblings, 6 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-14 13:16 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
The native path may be too big, leading to shebang
overflow. Just use the #!/usr/bin/env python3.
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
...shebang-overflow-on-python-config.py.patch | 33 +++++++++++++++++++
.../recipes-devtools/python/python3_3.10.4.bb | 1 +
2 files changed, 34 insertions(+)
create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
new file mode 100644
index 0000000000..921da8de7c
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
@@ -0,0 +1,33 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <ptsneves@gmail.com>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Denial reason: [1]
+
+Upstream-Status: Denied [distribution]
+
+[1] https://github.com/python/cpython/pull/93760#pullrequestreview-1005365737
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+ @ # Substitution happens here, as the completely-expanded BINDIR
+ @ # is not available in configure
+ sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++ @ # Otherwise we might get huge shebangs with native paths
++ sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+ @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+ LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+ @ # In OpenEmbedded, always use the python version of the script, the shell
+--
+2.25.1
+
diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
index 6bd3a6aba8..357025f856 100644
--- a/meta/recipes-devtools/python/python3_3.10.4.bb
+++ b/meta/recipes-devtools/python/python3_3.10.4.bb
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
file://deterministic_imports.patch \
+ file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
"
SRC_URI:append:class-native = " \
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v4 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2
2022-06-14 13:16 ` [PATCH v4 " Paulo Neves
@ 2022-06-14 13:16 ` Paulo Neves
2022-06-14 13:16 ` [PATCH v4 3/7] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
` (4 subsequent siblings)
5 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-14 13:16 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
A native path can cause a shebang overflow on gtkdoc-mkhtml.
Replace it with /usr/bin/env.
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
index 150d2c0b23..392913fcc6 100644
--- a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
+++ b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
@@ -43,6 +43,7 @@ do_install:append () {
${datadir}/gtk-doc/python/gtkdoc/config.py; do
sed -e 's,${RECIPE_SYSROOT_NATIVE}/usr/bin/pkg-config,${bindir}/pkg-config,' \
-e 's,${HOSTTOOLS_DIR}/python3,${bindir}/python3,' \
+ -e '1s|^#!.*|#!/usr/bin/env python3|' \
-i ${D}$fn
done
}
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v4 3/7] insane.bbclass: Make do_qa_staging check shebangs
2022-06-14 13:16 ` [PATCH v4 " Paulo Neves
2022-06-14 13:16 ` [PATCH v4 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2 Paulo Neves
@ 2022-06-14 13:16 ` Paulo Neves
2022-06-14 13:16 ` [PATCH v4 4/7] oeqa/selftest: Add test for shebang overflow Paulo Neves
` (3 subsequent siblings)
5 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-14 13:16 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
As reported in the bug report [1], there was no check for shebang
sizes on native scripts and now this is fixed.
The path scope of the qa_staging was increased from just checking
libdir to all the relevant SYSROOT_DIRS.
It is possible to skip this check through INSANE_SKIP.
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
meta/classes/insane.bbclass | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 9ca84bace9..b2951a48fe 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
skip_pkgconfig = True
+ skip_shebang_size = False
+ if 'shebang-size' in skip:
+ bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+ skip_shebang_size = True
+
# find all .la and .pc files
# read the content
# and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
oe.qa.handle_error("pkgconfig", error_msg, d)
+ if not skip_shebang_size:
+ errors = {}
+ package_qa_check_shebang_size(path, "", d, None, errors)
+ for e in errors:
+ oe.qa.handle_error(e, errors[e], d)
+
+
# Run all package-wide warnfuncs and errorfuncs
def package_qa_package(warnfuncs, errorfuncs, package, d):
warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
python do_qa_staging() {
bb.note("QA checking staging")
- qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+ sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+ for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+ qa_check_staged(sysroot_destdir + sysroot_dir, d)
oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
}
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v4 4/7] oeqa/selftest: Add test for shebang overflow
2022-06-14 13:16 ` [PATCH v4 " Paulo Neves
2022-06-14 13:16 ` [PATCH v4 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2 Paulo Neves
2022-06-14 13:16 ` [PATCH v4 3/7] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
@ 2022-06-14 13:16 ` Paulo Neves
2022-06-14 13:16 ` [PATCH v4 5/7] oeqa/selftest: Test staged .la and .pc files Paulo Neves
` (2 subsequent siblings)
5 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-14 13:16 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
Make sure we do not stage any executable with a bigger shebang
than 128. Fixes [1]
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
.../sysroot-test/sysroot-shebang-test_1.0.bb | 12 ++++++++++++
meta/lib/oeqa/selftest/cases/sysroot.py | 10 ++++++++++
2 files changed, 22 insertions(+)
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
new file mode 100644
index 0000000000..6c834be897
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+ install -d ${D}${bindir}
+ echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+ chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 315d1a61c2..9457f1e3ac 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -35,3 +35,13 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
TESTSTRING:pn-sysroot-test-arch2 = "%s"
""" % (uuid1, uuid2))
bitbake("sysroot-test")
+
+ def test_sysroot_max_shebang(self):
+ """
+ Summary: Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+ Expected: Fail when a shebang bigger than the max shebang-size is reached.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+ res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v4 5/7] oeqa/selftest: Test staged .la and .pc files
2022-06-14 13:16 ` [PATCH v4 " Paulo Neves
` (2 preceding siblings ...)
2022-06-14 13:16 ` [PATCH v4 4/7] oeqa/selftest: Add test for shebang overflow Paulo Neves
@ 2022-06-14 13:16 ` Paulo Neves
2022-06-14 13:16 ` [PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util Paulo Neves
2022-06-14 13:16 ` [PATCH v4 7/7] libcheck: Fix too long shebang for native case Paulo Neves
5 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-14 13:16 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
These files are checked by qa_check_staged but there was no
test cases for whether the tests actually worked. Now there
are.
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
.../sysroot-test/sysroot-la-test_1.0.bb | 16 ++++++++++
.../sysroot-test/sysroot-pc-test_1.0.bb | 12 +++++++
meta/lib/oeqa/selftest/cases/sysroot.py | 32 +++++++++++++++++++
3 files changed, 60 insertions(+)
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
new file mode 100644
index 0000000000..21f06782fb
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+ install -d ${D}${libdir}/test/
+ echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
new file mode 100644
index 0000000000..e748310fc4
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+ install -d ${D}${libdir}/test/
+ echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 9457f1e3ac..588fc8c713 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -45,3 +45,35 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
self.assertTrue(expected in res.output, msg=res.output)
+
+ def test_sysroot_la(self):
+ """
+ Summary: Check that workdir paths are not contained in .la files.
+ Expected: Fail when a workdir path is found in the file content.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "la-test.la failed sanity test (workdir) in path"
+
+ res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[la]' in res.output, msg=res.output)
+
+ res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[la]' in res.output, msg=res.output)
+
+ def test_sysroot_pkgconfig(self):
+ """
+ Summary: Check that tmpdir paths are not contained in .pc files.
+ Expected: Fail when a tmpdir path is found in the file content.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "test.pc failed sanity test (tmpdir) in path"
+
+ res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+ self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+ self.assertTrue(expected in res.output, msg=res.output)
+
+ res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util.
2022-06-14 13:16 ` [PATCH v4 " Paulo Neves
` (3 preceding siblings ...)
2022-06-14 13:16 ` [PATCH v4 5/7] oeqa/selftest: Test staged .la and .pc files Paulo Neves
@ 2022-06-14 13:16 ` Paulo Neves
2022-06-14 13:39 ` [OE-core] " Martin Jansa
2022-06-14 13:16 ` [PATCH v4 7/7] libcheck: Fix too long shebang for native case Paulo Neves
5 siblings, 1 reply; 64+ messages in thread
From: Paulo Neves @ 2022-06-14 13:16 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
Useful to work around shebang relocation issues, where
shebangs are too long or have arguments in them, thus preventing them
from using the /usr/bin/env shebang.
---
.../wrapper/cmdline-shebang-wrapper-test.bb | 21 ++++++++++++
.../recipes-test/wrapper/files/test.awk | 2 ++
meta/classes/utils.bbclass | 34 +++++++++++++++++++
meta/lib/oeqa/selftest/cases/wrapper.py | 11 ++++++
4 files changed, 68 insertions(+)
create mode 100644 meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
diff --git a/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
new file mode 100644
index 0000000000..302eea8901
--- /dev/null
+++ b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
@@ -0,0 +1,21 @@
+FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
+SUMMARY = "Check that create_cmdline_shebang works"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI += "file://test.awk"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+ install -d ${D}${bindir}
+ install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
+ sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
+ create_cmdline_shebang_wrapper ${D}${bindir}/test
+ if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
+ bbfatal "Wrapper is broken"
+ else
+ bbnote "Wrapper is good"
+ fi
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk b/meta-selftest/recipes-test/wrapper/files/test.awk
new file mode 100644
index 0000000000..91429197b1
--- /dev/null
+++ b/meta-selftest/recipes-test/wrapper/files/test.awk
@@ -0,0 +1,2 @@
+#! @AWK_BIN@ -f
+BEGIN { print "Don't Panic!" }
diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
index b4eb3d38ab..b617632d9f 100644
--- a/meta/classes/utils.bbclass
+++ b/meta/classes/utils.bbclass
@@ -184,6 +184,40 @@ END
chmod +x $cmd
}
+create_cmdline_shebang_wrapper () {
+ # Create a wrapper script where commandline options are needed
+ #
+ # These are useful to work around shebang relocation issues, where shebangs are too
+ # long or have arguments in them, thus preventing them from using the /usr/bin/env
+ # shebang
+ #
+ # Usage: create_cmdline_wrapper FILENAME <extra-options>
+
+ cmd=$1
+ shift
+
+ echo "Generating wrapper script for $cmd"
+
+ # Strip #! and get remaining interpreter + arg
+ argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
+ # strip the shebang from the real script as we do not want it to be usable anyway
+ tail -n +2 $cmd > $cmd.real
+ cmdname=$(basename $cmd)
+ dirname=$(dirname $cmd)
+ cmdoptions=$@
+ if [ "${base_prefix}" != "" ]; then
+ relpath=`python3 -c "import os; print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
+ cmdoptions=`echo $@ | sed -e "s:${base_prefix}:\\$realdir/$relpath:g"`
+ fi
+ cat <<END >$cmd
+#!/usr/bin/env bash
+realpath=\`readlink -fn \$0\`
+realdir=\`dirname \$realpath\`
+exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions "\$@"
+END
+ chmod +x $cmd
+}
+
create_wrapper () {
# Create a wrapper script where extra environment variables are needed
#
diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py b/meta/lib/oeqa/selftest/cases/wrapper.py
new file mode 100644
index 0000000000..6de63310c0
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/wrapper.py
@@ -0,0 +1,11 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake
+
+class WrapperTests(OESelftestTestCase):
+ def test_shebang_wrapper(self):
+ """
+ Summary: Build a recipe which will fail if the cmdline_shebang_wrapper function is defective.
+ Expected: Exit status to be 0.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ res = bitbake("cmdline-shebang-wrapper-test -c install", ignore_status=False)
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util.
2022-06-14 13:16 ` [PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util Paulo Neves
@ 2022-06-14 13:39 ` Martin Jansa
2022-06-14 13:46 ` Paulo Neves
0 siblings, 1 reply; 64+ messages in thread
From: Martin Jansa @ 2022-06-14 13:39 UTC (permalink / raw)
To: Paulo Neves; +Cc: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 5050 bytes --]
FILESEXTRAPATHS:prepend doesn't seem to be needed and LICENSE is really
CLOSED?
On Tue, Jun 14, 2022 at 3:17 PM Paulo Neves <ptsneves@gmail.com> wrote:
> Useful to work around shebang relocation issues, where
> shebangs are too long or have arguments in them, thus preventing them
> from using the /usr/bin/env shebang.
> ---
> .../wrapper/cmdline-shebang-wrapper-test.bb | 21 ++++++++++++
> .../recipes-test/wrapper/files/test.awk | 2 ++
> meta/classes/utils.bbclass | 34 +++++++++++++++++++
> meta/lib/oeqa/selftest/cases/wrapper.py | 11 ++++++
> 4 files changed, 68 insertions(+)
> create mode 100644 meta-selftest/recipes-test/wrapper/
> cmdline-shebang-wrapper-test.bb
> create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
> create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
>
> diff --git a/meta-selftest/recipes-test/wrapper/
> cmdline-shebang-wrapper-test.bb b/meta-selftest/recipes-test/wrapper/
> cmdline-shebang-wrapper-test.bb
> new file mode 100644
> index 0000000000..302eea8901
> --- /dev/null
> +++ b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
> @@ -0,0 +1,21 @@
> +FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
> +SUMMARY = "Check that create_cmdline_shebang works"
> +LICENSE = "CLOSED"
> +INHIBIT_DEFAULT_DEPS = "1"
> +
> +SRC_URI += "file://test.awk"
> +
> +EXCLUDE_FROM_WORLD = "1"
> +do_install() {
> + install -d ${D}${bindir}
> + install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
> + sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
> + create_cmdline_shebang_wrapper ${D}${bindir}/test
> + if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
> + bbfatal "Wrapper is broken"
> + else
> + bbnote "Wrapper is good"
> + fi
> +}
> +
> +BBCLASSEXTEND = "native"
> diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk
> b/meta-selftest/recipes-test/wrapper/files/test.awk
> new file mode 100644
> index 0000000000..91429197b1
> --- /dev/null
> +++ b/meta-selftest/recipes-test/wrapper/files/test.awk
> @@ -0,0 +1,2 @@
> +#! @AWK_BIN@ -f
> +BEGIN { print "Don't Panic!" }
> diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
> index b4eb3d38ab..b617632d9f 100644
> --- a/meta/classes/utils.bbclass
> +++ b/meta/classes/utils.bbclass
> @@ -184,6 +184,40 @@ END
> chmod +x $cmd
> }
>
> +create_cmdline_shebang_wrapper () {
> + # Create a wrapper script where commandline options are needed
> + #
> + # These are useful to work around shebang relocation issues, where
> shebangs are too
> + # long or have arguments in them, thus preventing them from using the
> /usr/bin/env
> + # shebang
> + #
> + # Usage: create_cmdline_wrapper FILENAME <extra-options>
> +
> + cmd=$1
> + shift
> +
> + echo "Generating wrapper script for $cmd"
> +
> + # Strip #! and get remaining interpreter + arg
> + argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
> + # strip the shebang from the real script as we do not want it to be
> usable anyway
> + tail -n +2 $cmd > $cmd.real
> + cmdname=$(basename $cmd)
> + dirname=$(dirname $cmd)
> + cmdoptions=$@
> + if [ "${base_prefix}" != "" ]; then
> + relpath=`python3 -c "import os;
> print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
> + cmdoptions=`echo $@ | sed -e
> "s:${base_prefix}:\\$realdir/$relpath:g"`
> + fi
> + cat <<END >$cmd
> +#!/usr/bin/env bash
> +realpath=\`readlink -fn \$0\`
> +realdir=\`dirname \$realpath\`
> +exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions
> "\$@"
> +END
> + chmod +x $cmd
> +}
> +
> create_wrapper () {
> # Create a wrapper script where extra environment variables are
> needed
> #
> diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py
> b/meta/lib/oeqa/selftest/cases/wrapper.py
> new file mode 100644
> index 0000000000..6de63310c0
> --- /dev/null
> +++ b/meta/lib/oeqa/selftest/cases/wrapper.py
> @@ -0,0 +1,11 @@
> +from oeqa.selftest.case import OESelftestTestCase
> +from oeqa.utils.commands import bitbake
> +
> +class WrapperTests(OESelftestTestCase):
> + def test_shebang_wrapper(self):
> + """
> + Summary: Build a recipe which will fail if the
> cmdline_shebang_wrapper function is defective.
> + Expected: Exit status to be 0.
> + Author: Paulo Neves <ptsneves@gmail.com>
> + """
> + res = bitbake("cmdline-shebang-wrapper-test -c install",
> ignore_status=False)
> --
> 2.25.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#166887):
> https://lists.openembedded.org/g/openembedded-core/message/166887
> Mute This Topic: https://lists.openembedded.org/mt/91748692/3617156
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> Martin.Jansa@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
[-- Attachment #2: Type: text/html, Size: 7092 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util.
2022-06-14 13:39 ` [OE-core] " Martin Jansa
@ 2022-06-14 13:46 ` Paulo Neves
2022-06-14 14:17 ` Martin Jansa
0 siblings, 1 reply; 64+ messages in thread
From: Paulo Neves @ 2022-06-14 13:46 UTC (permalink / raw)
To: Martin Jansa; +Cc: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 6066 bytes --]
Regarding the FILESEXTRAPATHS:prepend i was convinced that
${THISDIR}/files was not automatically added and was needed for the
test.awk file. I guess I have been cargo culting for some time on this.
Will remove it.
This is a recipe for a test and if i add a license different than CLOSED
than i need to add a license file, which seems a bit overkill for a one
line. Am I wrong? What license should it even be? Dont Panic is a common
string used in awk docs.
Paulo Neves
On 6/14/22 15:39, Martin Jansa wrote:
> FILESEXTRAPATHS:prepend doesn't seem to be needed and LICENSE is
> really CLOSED?
>
> On Tue, Jun 14, 2022 at 3:17 PM Paulo Neves <ptsneves@gmail.com> wrote:
>
> Useful to work around shebang relocation issues, where
> shebangs are too long or have arguments in them, thus preventing them
> from using the /usr/bin/env shebang.
> ---
> .../wrapper/cmdline-shebang-wrapper-test.bb
> <http://cmdline-shebang-wrapper-test.bb> | 21 ++++++++++++
> .../recipes-test/wrapper/files/test.awk | 2 ++
> meta/classes/utils.bbclass | 34
> +++++++++++++++++++
> meta/lib/oeqa/selftest/cases/wrapper.py | 11 ++++++
> 4 files changed, 68 insertions(+)
> create mode 100644
> meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
> <http://cmdline-shebang-wrapper-test.bb>
> create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
> create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
>
> diff --git
> a/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
> <http://cmdline-shebang-wrapper-test.bb>
> b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
> <http://cmdline-shebang-wrapper-test.bb>
> new file mode 100644
> index 0000000000..302eea8901
> --- /dev/null
> +++
> b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
> <http://cmdline-shebang-wrapper-test.bb>
> @@ -0,0 +1,21 @@
> +FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
> +SUMMARY = "Check that create_cmdline_shebang works"
> +LICENSE = "CLOSED"
> +INHIBIT_DEFAULT_DEPS = "1"
> +
> +SRC_URI += "file://test.awk"
> +
> +EXCLUDE_FROM_WORLD = "1"
> +do_install() {
> + install -d ${D}${bindir}
> + install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
> + sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
> + create_cmdline_shebang_wrapper ${D}${bindir}/test
> + if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
> + bbfatal "Wrapper is broken"
> + else
> + bbnote "Wrapper is good"
> + fi
> +}
> +
> +BBCLASSEXTEND = "native"
> diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk
> b/meta-selftest/recipes-test/wrapper/files/test.awk
> new file mode 100644
> index 0000000000..91429197b1
> --- /dev/null
> +++ b/meta-selftest/recipes-test/wrapper/files/test.awk
> @@ -0,0 +1,2 @@
> +#! @AWK_BIN@ -f
> +BEGIN { print "Don't Panic!" }
> diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
> index b4eb3d38ab..b617632d9f 100644
> --- a/meta/classes/utils.bbclass
> +++ b/meta/classes/utils.bbclass
> @@ -184,6 +184,40 @@ END
> chmod +x $cmd
> }
>
> +create_cmdline_shebang_wrapper () {
> + # Create a wrapper script where commandline options are needed
> + #
> + # These are useful to work around shebang relocation
> issues, where shebangs are too
> + # long or have arguments in them, thus preventing them from
> using the /usr/bin/env
> + # shebang
> + #
> + # Usage: create_cmdline_wrapper FILENAME <extra-options>
> +
> + cmd=$1
> + shift
> +
> + echo "Generating wrapper script for $cmd"
> +
> + # Strip #! and get remaining interpreter + arg
> + argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
> + # strip the shebang from the real script as we do not want it
> to be usable anyway
> + tail -n +2 $cmd > $cmd.real
> + cmdname=$(basename $cmd)
> + dirname=$(dirname $cmd)
> + cmdoptions=$@
> + if [ "${base_prefix}" != "" ]; then
> + relpath=`python3 -c "import os;
> print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
> + cmdoptions=`echo $@ | sed -e
> "s:${base_prefix}:\\$realdir/$relpath:g"`
> + fi
> + cat <<END >$cmd
> +#!/usr/bin/env bash
> +realpath=\`readlink -fn \$0\`
> +realdir=\`dirname \$realpath\`
> +exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real
> $cmdoptions "\$@"
> +END
> + chmod +x $cmd
> +}
> +
> create_wrapper () {
> # Create a wrapper script where extra environment
> variables are needed
> #
> diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py
> b/meta/lib/oeqa/selftest/cases/wrapper.py
> new file mode 100644
> index 0000000000..6de63310c0
> --- /dev/null
> +++ b/meta/lib/oeqa/selftest/cases/wrapper.py
> @@ -0,0 +1,11 @@
> +from oeqa.selftest.case import OESelftestTestCase
> +from oeqa.utils.commands import bitbake
> +
> +class WrapperTests(OESelftestTestCase):
> + def test_shebang_wrapper(self):
> + """
> + Summary: Build a recipe which will fail if the
> cmdline_shebang_wrapper function is defective.
> + Expected: Exit status to be 0.
> + Author: Paulo Neves <ptsneves@gmail.com>
> + """
> + res = bitbake("cmdline-shebang-wrapper-test -c install",
> ignore_status=False)
> --
> 2.25.1
>
>
>
>
[-- Attachment #2: Type: text/html, Size: 8661 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util.
2022-06-14 13:46 ` Paulo Neves
@ 2022-06-14 14:17 ` Martin Jansa
2022-06-14 15:12 ` Paulo Neves
0 siblings, 1 reply; 64+ messages in thread
From: Martin Jansa @ 2022-06-14 14:17 UTC (permalink / raw)
To: Paulo Neves; +Cc: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 6292 bytes --]
You can use bitbake-getvar to see the default FILESPATH, but "files" next
to recipe is included by default (I prefer to use BPN directory for
slightly faster lookup - unless the files are shared between different
recipes). FILESEXTRAPATHS is usually only needed from bbappends which add
new files to SRC_URI.
For LICENSE you can use MIT and
LIC_FILES_CHKSUM =
"file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
like many "pure-metadata" recipes in oe-core.
On Tue, Jun 14, 2022 at 3:46 PM Paulo Neves <ptsneves@gmail.com> wrote:
> Regarding the FILESEXTRAPATHS:prepend i was convinced that
> ${THISDIR}/files was not automatically added and was needed for the
> test.awk file. I guess I have been cargo culting for some time on this.
> Will remove it.
>
> This is a recipe for a test and if i add a license different than CLOSED
> than i need to add a license file, which seems a bit overkill for a one
> line. Am I wrong? What license should it even be? Dont Panic is a common
> string used in awk docs.
>
> Paulo Neves
>
> On 6/14/22 15:39, Martin Jansa wrote:
>
> FILESEXTRAPATHS:prepend doesn't seem to be needed and LICENSE is really
> CLOSED?
>
> On Tue, Jun 14, 2022 at 3:17 PM Paulo Neves <ptsneves@gmail.com> wrote:
>
>> Useful to work around shebang relocation issues, where
>> shebangs are too long or have arguments in them, thus preventing them
>> from using the /usr/bin/env shebang.
>> ---
>> .../wrapper/cmdline-shebang-wrapper-test.bb | 21 ++++++++++++
>> .../recipes-test/wrapper/files/test.awk | 2 ++
>> meta/classes/utils.bbclass | 34 +++++++++++++++++++
>> meta/lib/oeqa/selftest/cases/wrapper.py | 11 ++++++
>> 4 files changed, 68 insertions(+)
>> create mode 100644 meta-selftest/recipes-test/wrapper/
>> cmdline-shebang-wrapper-test.bb
>> create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
>> create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
>>
>> diff --git a/meta-selftest/recipes-test/wrapper/
>> cmdline-shebang-wrapper-test.bb b/meta-selftest/recipes-test/wrapper/
>> cmdline-shebang-wrapper-test.bb
>> new file mode 100644
>> index 0000000000..302eea8901
>> --- /dev/null
>> +++ b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
>> @@ -0,0 +1,21 @@
>> +FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
>> +SUMMARY = "Check that create_cmdline_shebang works"
>> +LICENSE = "CLOSED"
>> +INHIBIT_DEFAULT_DEPS = "1"
>> +
>> +SRC_URI += "file://test.awk"
>> +
>> +EXCLUDE_FROM_WORLD = "1"
>> +do_install() {
>> + install -d ${D}${bindir}
>> + install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
>> + sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
>> + create_cmdline_shebang_wrapper ${D}${bindir}/test
>> + if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
>> + bbfatal "Wrapper is broken"
>> + else
>> + bbnote "Wrapper is good"
>> + fi
>> +}
>> +
>> +BBCLASSEXTEND = "native"
>> diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk
>> b/meta-selftest/recipes-test/wrapper/files/test.awk
>> new file mode 100644
>> index 0000000000..91429197b1
>> --- /dev/null
>> +++ b/meta-selftest/recipes-test/wrapper/files/test.awk
>> @@ -0,0 +1,2 @@
>> +#! @AWK_BIN@ -f
>> +BEGIN { print "Don't Panic!" }
>> diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
>> index b4eb3d38ab..b617632d9f 100644
>> --- a/meta/classes/utils.bbclass
>> +++ b/meta/classes/utils.bbclass
>> @@ -184,6 +184,40 @@ END
>> chmod +x $cmd
>> }
>>
>> +create_cmdline_shebang_wrapper () {
>> + # Create a wrapper script where commandline options are needed
>> + #
>> + # These are useful to work around shebang relocation issues,
>> where shebangs are too
>> + # long or have arguments in them, thus preventing them from using the
>> /usr/bin/env
>> + # shebang
>> + #
>> + # Usage: create_cmdline_wrapper FILENAME <extra-options>
>> +
>> + cmd=$1
>> + shift
>> +
>> + echo "Generating wrapper script for $cmd"
>> +
>> + # Strip #! and get remaining interpreter + arg
>> + argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
>> + # strip the shebang from the real script as we do not want it to be
>> usable anyway
>> + tail -n +2 $cmd > $cmd.real
>> + cmdname=$(basename $cmd)
>> + dirname=$(dirname $cmd)
>> + cmdoptions=$@
>> + if [ "${base_prefix}" != "" ]; then
>> + relpath=`python3 -c "import os;
>> print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
>> + cmdoptions=`echo $@ | sed -e
>> "s:${base_prefix}:\\$realdir/$relpath:g"`
>> + fi
>> + cat <<END >$cmd
>> +#!/usr/bin/env bash
>> +realpath=\`readlink -fn \$0\`
>> +realdir=\`dirname \$realpath\`
>> +exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions
>> "\$@"
>> +END
>> + chmod +x $cmd
>> +}
>> +
>> create_wrapper () {
>> # Create a wrapper script where extra environment variables are
>> needed
>> #
>> diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py
>> b/meta/lib/oeqa/selftest/cases/wrapper.py
>> new file mode 100644
>> index 0000000000..6de63310c0
>> --- /dev/null
>> +++ b/meta/lib/oeqa/selftest/cases/wrapper.py
>> @@ -0,0 +1,11 @@
>> +from oeqa.selftest.case import OESelftestTestCase
>> +from oeqa.utils.commands import bitbake
>> +
>> +class WrapperTests(OESelftestTestCase):
>> + def test_shebang_wrapper(self):
>> + """
>> + Summary: Build a recipe which will fail if the
>> cmdline_shebang_wrapper function is defective.
>> + Expected: Exit status to be 0.
>> + Author: Paulo Neves <ptsneves@gmail.com>
>> + """
>> + res = bitbake("cmdline-shebang-wrapper-test -c install",
>> ignore_status=False)
>> --
>> 2.25.1
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#166887):
>> https://lists.openembedded.org/g/openembedded-core/message/166887
>> Mute This Topic: https://lists.openembedded.org/mt/91748692/3617156
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
>> Martin.Jansa@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
>>
>
[-- Attachment #2: Type: text/html, Size: 10324 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util.
2022-06-14 14:17 ` Martin Jansa
@ 2022-06-14 15:12 ` Paulo Neves
0 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-14 15:12 UTC (permalink / raw)
To: Martin Jansa; +Cc: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 7685 bytes --]
Thanks for the great tips. Yeah the habit of using the FILESPATH is that
mostly I work on bbappends. Today i learned.
I sent a new v5 with the corrections you mention.
Paulo Neves
On 6/14/22 16:17, Martin Jansa wrote:
> You can use bitbake-getvar to see the default FILESPATH, but "files"
> next to recipe is included by default (I prefer to use BPN directory
> for slightly faster lookup - unless the files are shared between
> different recipes). FILESEXTRAPATHS is usually only needed from
> bbappends which add new files to SRC_URI.
>
> For LICENSE you can use MIT and
> LIC_FILES_CHKSUM =
> "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
> like many "pure-metadata" recipes in oe-core.
>
> On Tue, Jun 14, 2022 at 3:46 PM Paulo Neves <ptsneves@gmail.com> wrote:
>
> Regarding the FILESEXTRAPATHS:prepend i was convinced that
> ${THISDIR}/files was not automatically added and was needed for
> the test.awk file. I guess I have been cargo culting for some time
> on this. Will remove it.
>
> This is a recipe for a test and if i add a license different than
> CLOSED than i need to add a license file, which seems a bit
> overkill for a one line. Am I wrong? What license should it even
> be? Dont Panic is a common string used in awk docs.
>
> Paulo Neves
>
> On 6/14/22 15:39, Martin Jansa wrote:
>> FILESEXTRAPATHS:prepend doesn't seem to be needed and LICENSE is
>> really CLOSED?
>>
>> On Tue, Jun 14, 2022 at 3:17 PM Paulo Neves <ptsneves@gmail.com>
>> wrote:
>>
>> Useful to work around shebang relocation issues, where
>> shebangs are too long or have arguments in them, thus
>> preventing them
>> from using the /usr/bin/env shebang.
>> ---
>> .../wrapper/cmdline-shebang-wrapper-test.bb
>> <http://cmdline-shebang-wrapper-test.bb> | 21 ++++++++++++
>> .../recipes-test/wrapper/files/test.awk | 2 ++
>> meta/classes/utils.bbclass | 34
>> +++++++++++++++++++
>> meta/lib/oeqa/selftest/cases/wrapper.py | 11 ++++++
>> 4 files changed, 68 insertions(+)
>> create mode 100644
>> meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
>> <http://cmdline-shebang-wrapper-test.bb>
>> create mode 100644
>> meta-selftest/recipes-test/wrapper/files/test.awk
>> create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
>>
>> diff --git
>> a/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
>> <http://cmdline-shebang-wrapper-test.bb>
>> b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
>> <http://cmdline-shebang-wrapper-test.bb>
>> new file mode 100644
>> index 0000000000..302eea8901
>> --- /dev/null
>> +++
>> b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
>> <http://cmdline-shebang-wrapper-test.bb>
>> @@ -0,0 +1,21 @@
>> +FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
>> +SUMMARY = "Check that create_cmdline_shebang works"
>> +LICENSE = "CLOSED"
>> +INHIBIT_DEFAULT_DEPS = "1"
>> +
>> +SRC_URI += "file://test.awk"
>> +
>> +EXCLUDE_FROM_WORLD = "1"
>> +do_install() {
>> + install -d ${D}${bindir}
>> + install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
>> + sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
>> + create_cmdline_shebang_wrapper ${D}${bindir}/test
>> + if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
>> + bbfatal "Wrapper is broken"
>> + else
>> + bbnote "Wrapper is good"
>> + fi
>> +}
>> +
>> +BBCLASSEXTEND = "native"
>> diff --git
>> a/meta-selftest/recipes-test/wrapper/files/test.awk
>> b/meta-selftest/recipes-test/wrapper/files/test.awk
>> new file mode 100644
>> index 0000000000..91429197b1
>> --- /dev/null
>> +++ b/meta-selftest/recipes-test/wrapper/files/test.awk
>> @@ -0,0 +1,2 @@
>> +#! @AWK_BIN@ -f
>> +BEGIN { print "Don't Panic!" }
>> diff --git a/meta/classes/utils.bbclass
>> b/meta/classes/utils.bbclass
>> index b4eb3d38ab..b617632d9f 100644
>> --- a/meta/classes/utils.bbclass
>> +++ b/meta/classes/utils.bbclass
>> @@ -184,6 +184,40 @@ END
>> chmod +x $cmd
>> }
>>
>> +create_cmdline_shebang_wrapper () {
>> + # Create a wrapper script where commandline options
>> are needed
>> + #
>> + # These are useful to work around shebang relocation
>> issues, where shebangs are too
>> + # long or have arguments in them, thus preventing them
>> from using the /usr/bin/env
>> + # shebang
>> + #
>> + # Usage: create_cmdline_wrapper FILENAME <extra-options>
>> +
>> + cmd=$1
>> + shift
>> +
>> + echo "Generating wrapper script for $cmd"
>> +
>> + # Strip #! and get remaining interpreter + arg
>> + argument="$(basename "$(head -n1 $cmd | sed -e 's|#![
>> ]*||g' )")"
>> + # strip the shebang from the real script as we do not want
>> it to be usable anyway
>> + tail -n +2 $cmd > $cmd.real
>> + cmdname=$(basename $cmd)
>> + dirname=$(dirname $cmd)
>> + cmdoptions=$@
>> + if [ "${base_prefix}" != "" ]; then
>> + relpath=`python3 -c "import os;
>> print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
>> + cmdoptions=`echo $@ | sed -e
>> "s:${base_prefix}:\\$realdir/$relpath:g"`
>> + fi
>> + cat <<END >$cmd
>> +#!/usr/bin/env bash
>> +realpath=\`readlink -fn \$0\`
>> +realdir=\`dirname \$realpath\`
>> +exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real
>> $cmdoptions "\$@"
>> +END
>> + chmod +x $cmd
>> +}
>> +
>> create_wrapper () {
>> # Create a wrapper script where extra environment
>> variables are needed
>> #
>> diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py
>> b/meta/lib/oeqa/selftest/cases/wrapper.py
>> new file mode 100644
>> index 0000000000..6de63310c0
>> --- /dev/null
>> +++ b/meta/lib/oeqa/selftest/cases/wrapper.py
>> @@ -0,0 +1,11 @@
>> +from oeqa.selftest.case import OESelftestTestCase
>> +from oeqa.utils.commands import bitbake
>> +
>> +class WrapperTests(OESelftestTestCase):
>> + def test_shebang_wrapper(self):
>> + """
>> + Summary: Build a recipe which will fail if the
>> cmdline_shebang_wrapper function is defective.
>> + Expected: Exit status to be 0.
>> + Author: Paulo Neves <ptsneves@gmail.com>
>> + """
>> + res = bitbake("cmdline-shebang-wrapper-test -c
>> install", ignore_status=False)
>> --
>> 2.25.1
>>
>>
>>
>>
>
[-- Attachment #2: Type: text/html, Size: 12012 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* [PATCH v4 7/7] libcheck: Fix too long shebang for native case.
2022-06-14 13:16 ` [PATCH v4 " Paulo Neves
` (4 preceding siblings ...)
2022-06-14 13:16 ` [PATCH v4 6/7] utils: Add cmdline_shebang_wrapper util Paulo Neves
@ 2022-06-14 13:16 ` Paulo Neves
5 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-14 13:16 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
It requires a shebang wrapper due to the fact that awk interpreter
has an argument.
---
meta/recipes-support/libcheck/libcheck_0.15.2.bb | 3 +++
1 file changed, 3 insertions(+)
diff --git a/meta/recipes-support/libcheck/libcheck_0.15.2.bb b/meta/recipes-support/libcheck/libcheck_0.15.2.bb
index 188d689cc3..1393aa2a1c 100644
--- a/meta/recipes-support/libcheck/libcheck_0.15.2.bb
+++ b/meta/recipes-support/libcheck/libcheck_0.15.2.bb
@@ -23,6 +23,9 @@ CACHED_CONFIGUREVARS += "ac_cv_path_AWK_PATH=${bindir}/gawk"
RREPLACES:${PN} = "check (<= 0.9.5)"
+do_install:append:class-native() {
+ create_cmdline_shebang_wrapper ${D}${bindir}/checkmk
+}
BBCLASSEXTEND = "native nativesdk"
PACKAGES =+ "checkmk"
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread
* [PATCH v5 1/7] python: Avoid shebang overflow on python-config.py
2022-06-06 15:58 [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
` (4 preceding siblings ...)
2022-06-14 13:16 ` [PATCH v4 " Paulo Neves
@ 2022-06-14 15:10 ` Paulo Neves
2022-06-14 15:11 ` [PATCH v5 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2 Paulo Neves
` (5 more replies)
2022-06-15 11:07 ` [OE-core] [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs Ross Burton
2022-06-19 19:20 ` [PATCH v6 1/7] python: Avoid shebang overflow on python-config.py Paulo Neves
7 siblings, 6 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-14 15:10 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
The native path may be too big, leading to shebang
overflow. Just use the #!/usr/bin/env python3.
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
...shebang-overflow-on-python-config.py.patch | 33 +++++++++++++++++++
.../recipes-devtools/python/python3_3.10.4.bb | 1 +
2 files changed, 34 insertions(+)
create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
new file mode 100644
index 0000000000..921da8de7c
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
@@ -0,0 +1,33 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <ptsneves@gmail.com>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Denial reason: [1]
+
+Upstream-Status: Denied [distribution]
+
+[1] https://github.com/python/cpython/pull/93760#pullrequestreview-1005365737
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+ @ # Substitution happens here, as the completely-expanded BINDIR
+ @ # is not available in configure
+ sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++ @ # Otherwise we might get huge shebangs with native paths
++ sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+ @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+ LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+ @ # In OpenEmbedded, always use the python version of the script, the shell
+--
+2.25.1
+
diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
index 6bd3a6aba8..357025f856 100644
--- a/meta/recipes-devtools/python/python3_3.10.4.bb
+++ b/meta/recipes-devtools/python/python3_3.10.4.bb
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
file://deterministic_imports.patch \
+ file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
"
SRC_URI:append:class-native = " \
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v5 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2
2022-06-14 15:10 ` [PATCH v5 1/7] python: Avoid shebang overflow on python-config.py Paulo Neves
@ 2022-06-14 15:11 ` Paulo Neves
2022-06-14 15:11 ` [PATCH v5 3/7] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
` (4 subsequent siblings)
5 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-14 15:11 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
A native path can cause a shebang overflow on gtkdoc-mkhtml.
Replace it with /usr/bin/env.
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
index 150d2c0b23..392913fcc6 100644
--- a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
+++ b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
@@ -43,6 +43,7 @@ do_install:append () {
${datadir}/gtk-doc/python/gtkdoc/config.py; do
sed -e 's,${RECIPE_SYSROOT_NATIVE}/usr/bin/pkg-config,${bindir}/pkg-config,' \
-e 's,${HOSTTOOLS_DIR}/python3,${bindir}/python3,' \
+ -e '1s|^#!.*|#!/usr/bin/env python3|' \
-i ${D}$fn
done
}
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v5 3/7] insane.bbclass: Make do_qa_staging check shebangs
2022-06-14 15:10 ` [PATCH v5 1/7] python: Avoid shebang overflow on python-config.py Paulo Neves
2022-06-14 15:11 ` [PATCH v5 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2 Paulo Neves
@ 2022-06-14 15:11 ` Paulo Neves
2022-06-17 16:50 ` [OE-core] " Richard Purdie
[not found] ` <16F97685B484595E.1329@lists.openembedded.org>
2022-06-14 15:11 ` [PATCH v5 4/7] oeqa/selftest: Add test for shebang overflow Paulo Neves
` (3 subsequent siblings)
5 siblings, 2 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-14 15:11 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
As reported in the bug report [1], there was no check for shebang
sizes on native scripts and now this is fixed.
The path scope of the qa_staging was increased from just checking
libdir to all the relevant SYSROOT_DIRS.
It is possible to skip this check through INSANE_SKIP.
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
meta/classes/insane.bbclass | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 9ca84bace9..b2951a48fe 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
skip_pkgconfig = True
+ skip_shebang_size = False
+ if 'shebang-size' in skip:
+ bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+ skip_shebang_size = True
+
# find all .la and .pc files
# read the content
# and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
oe.qa.handle_error("pkgconfig", error_msg, d)
+ if not skip_shebang_size:
+ errors = {}
+ package_qa_check_shebang_size(path, "", d, None, errors)
+ for e in errors:
+ oe.qa.handle_error(e, errors[e], d)
+
+
# Run all package-wide warnfuncs and errorfuncs
def package_qa_package(warnfuncs, errorfuncs, package, d):
warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
python do_qa_staging() {
bb.note("QA checking staging")
- qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+ sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+ for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+ qa_check_staged(sysroot_destdir + sysroot_dir, d)
oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
}
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v5 3/7] insane.bbclass: Make do_qa_staging check shebangs
2022-06-14 15:11 ` [PATCH v5 3/7] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
@ 2022-06-17 16:50 ` Richard Purdie
2022-06-19 10:53 ` Paulo Neves
[not found] ` <16F97685B484595E.1329@lists.openembedded.org>
1 sibling, 1 reply; 64+ messages in thread
From: Richard Purdie @ 2022-06-17 16:50 UTC (permalink / raw)
To: Paulo Neves, openembedded-core
On Tue, 2022-06-14 at 17:11 +0200, Paulo Neves wrote:
> As reported in the bug report [1], there was no check for shebang
> sizes on native scripts and now this is fixed.
>
> The path scope of the qa_staging was increased from just checking
> libdir to all the relevant SYSROOT_DIRS.
>
> It is possible to skip this check through INSANE_SKIP.
>
> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
>
> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> ---
> meta/classes/insane.bbclass | 16 +++++++++++++++-
> 1 file changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
> index 9ca84bace9..b2951a48fe 100644
> --- a/meta/classes/insane.bbclass
> +++ b/meta/classes/insane.bbclass
> @@ -630,6 +630,11 @@ def qa_check_staged(path,d):
> bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
> skip_pkgconfig = True
>
> + skip_shebang_size = False
> + if 'shebang-size' in skip:
> + bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
> + skip_shebang_size = True
> +
> # find all .la and .pc files
> # read the content
> # and check for stuff that looks wrong
> @@ -651,6 +656,13 @@ def qa_check_staged(path,d):
> error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
> oe.qa.handle_error("pkgconfig", error_msg, d)
>
> + if not skip_shebang_size:
> + errors = {}
> + package_qa_check_shebang_size(path, "", d, None, errors)
> + for e in errors:
> + oe.qa.handle_error(e, errors[e], d)
> +
> +
> # Run all package-wide warnfuncs and errorfuncs
> def package_qa_package(warnfuncs, errorfuncs, package, d):
> warnings = {}
> @@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
>
> python do_qa_staging() {
> bb.note("QA checking staging")
> - qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
> + sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
> + for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
> + qa_check_staged(sysroot_destdir + sysroot_dir, d)
> oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
> }
I'm a little worried about the performance implications of this, we're
going from scanning files in libdir to scanning nearly all files in the
sysroots, reading from many of them. In isolation that doesn't seem
much but I suspect the IO will impact builds overall.
That leaves me a little torn on this change :/
Cheers,
Richard
^ permalink raw reply [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v5 3/7] insane.bbclass: Make do_qa_staging check shebangs
2022-06-17 16:50 ` [OE-core] " Richard Purdie
@ 2022-06-19 10:53 ` Paulo Neves
0 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-19 10:53 UTC (permalink / raw)
To: Richard Purdie; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 3139 bytes --]
Yes, when I was making the change I also realized the scope of the check
was quite bigger than before. I am going to deliver the final changes
requested and leave it up to you to decide on the merge. Let me know your
decision so I can close the bug as well.
Paulo Neves
On Fri, Jun 17, 2022, 18:50 Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2022-06-14 at 17:11 +0200, Paulo Neves wrote:
> > As reported in the bug report [1], there was no check for shebang
> > sizes on native scripts and now this is fixed.
> >
> > The path scope of the qa_staging was increased from just checking
> > libdir to all the relevant SYSROOT_DIRS.
> >
> > It is possible to skip this check through INSANE_SKIP.
> >
> > [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
> >
> > Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> > ---
> > meta/classes/insane.bbclass | 16 +++++++++++++++-
> > 1 file changed, 15 insertions(+), 1 deletion(-)
> >
> > diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
> > index 9ca84bace9..b2951a48fe 100644
> > --- a/meta/classes/insane.bbclass
> > +++ b/meta/classes/insane.bbclass
> > @@ -630,6 +630,11 @@ def qa_check_staged(path,d):
> > bb.note("Recipe %s skipping qa checking: pkgconfig" %
> d.getVar('PN'))
> > skip_pkgconfig = True
> >
> > + skip_shebang_size = False
> > + if 'shebang-size' in skip:
> > + bb.note("Recipe %s skipping qa checkking: shebang-size" %
> d.getVar('PN'))
> > + skip_shebang_size = True
> > +
> > # find all .la and .pc files
> > # read the content
> > # and check for stuff that looks wrong
> > @@ -651,6 +656,13 @@ def qa_check_staged(path,d):
> > error_msg = "%s failed sanity test (tmpdir) in
> path %s" % (file,root)
> > oe.qa.handle_error("pkgconfig", error_msg, d)
> >
> > + if not skip_shebang_size:
> > + errors = {}
> > + package_qa_check_shebang_size(path, "", d, None, errors)
> > + for e in errors:
> > + oe.qa.handle_error(e, errors[e], d)
> > +
> > +
> > # Run all package-wide warnfuncs and errorfuncs
> > def package_qa_package(warnfuncs, errorfuncs, package, d):
> > warnings = {}
> > @@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
> >
> > python do_qa_staging() {
> > bb.note("QA checking staging")
> > - qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
> > + sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
> > + for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
> > + qa_check_staged(sysroot_destdir + sysroot_dir, d)
> > oe.qa.exit_with_message_if_errors("QA staging was broken by the
> package built above", d)
> > }
>
>
> I'm a little worried about the performance implications of this, we're
> going from scanning files in libdir to scanning nearly all files in the
> sysroots, reading from many of them. In isolation that doesn't seem
> much but I suspect the IO will impact builds overall.
>
> That leaves me a little torn on this change :/
>
> Cheers,
>
> Richard
>
>
[-- Attachment #2: Type: text/html, Size: 4289 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
[parent not found: <16F97685B484595E.1329@lists.openembedded.org>]
* Re: [OE-core] [PATCH v5 3/7] insane.bbclass: Make do_qa_staging check shebangs
[not found] ` <16F97685B484595E.1329@lists.openembedded.org>
@ 2022-07-01 12:53 ` Richard Purdie
0 siblings, 0 replies; 64+ messages in thread
From: Richard Purdie @ 2022-07-01 12:53 UTC (permalink / raw)
To: Paulo Neves, openembedded-core
On Fri, 2022-06-17 at 17:50 +0100, Richard Purdie via
lists.openembedded.org wrote:
> On Tue, 2022-06-14 at 17:11 +0200, Paulo Neves wrote:
> > As reported in the bug report [1], there was no check for shebang
> > sizes on native scripts and now this is fixed.
> >
> > The path scope of the qa_staging was increased from just checking
> > libdir to all the relevant SYSROOT_DIRS.
> >
> > It is possible to skip this check through INSANE_SKIP.
> >
> > [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
> >
> > Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> > ---
> > meta/classes/insane.bbclass | 16 +++++++++++++++-
> > 1 file changed, 15 insertions(+), 1 deletion(-)
> >
> > diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
> > index 9ca84bace9..b2951a48fe 100644
> > --- a/meta/classes/insane.bbclass
> > +++ b/meta/classes/insane.bbclass
> > @@ -630,6 +630,11 @@ def qa_check_staged(path,d):
> > bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
> > skip_pkgconfig = True
> >
> > + skip_shebang_size = False
> > + if 'shebang-size' in skip:
> > + bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
> > + skip_shebang_size = True
> > +
> > # find all .la and .pc files
> > # read the content
> > # and check for stuff that looks wrong
> > @@ -651,6 +656,13 @@ def qa_check_staged(path,d):
> > error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
> > oe.qa.handle_error("pkgconfig", error_msg, d)
> >
> > + if not skip_shebang_size:
> > + errors = {}
> > + package_qa_check_shebang_size(path, "", d, None, errors)
> > + for e in errors:
> > + oe.qa.handle_error(e, errors[e], d)
> > +
> > +
> > # Run all package-wide warnfuncs and errorfuncs
> > def package_qa_package(warnfuncs, errorfuncs, package, d):
> > warnings = {}
> > @@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
> >
> > python do_qa_staging() {
> > bb.note("QA checking staging")
> > - qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
> > + sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
> > + for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
> > + qa_check_staged(sysroot_destdir + sysroot_dir, d)
> > oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
> > }
>
>
> I'm a little worried about the performance implications of this, we're
> going from scanning files in libdir to scanning nearly all files in the
> sysroots, reading from many of them. In isolation that doesn't seem
> much but I suspect the IO will impact builds overall.
>
> That leaves me a little torn on this change :/
Sorry about the delay on this. My concern isn't an issue as I wasn't
remembering the exact context of this code correctly. The patches have
therefore merged, thanks!
Cheers,
Richard
^ permalink raw reply [flat|nested] 64+ messages in thread
* [PATCH v5 4/7] oeqa/selftest: Add test for shebang overflow
2022-06-14 15:10 ` [PATCH v5 1/7] python: Avoid shebang overflow on python-config.py Paulo Neves
2022-06-14 15:11 ` [PATCH v5 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2 Paulo Neves
2022-06-14 15:11 ` [PATCH v5 3/7] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
@ 2022-06-14 15:11 ` Paulo Neves
2022-06-14 15:11 ` [PATCH v5 5/7] oeqa/selftest: Test staged .la and .pc files Paulo Neves
` (2 subsequent siblings)
5 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-14 15:11 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
Make sure we do not stage any executable with a bigger shebang
than 128. Fixes [1]
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
.../sysroot-test/sysroot-shebang-test_1.0.bb | 12 ++++++++++++
meta/lib/oeqa/selftest/cases/sysroot.py | 10 ++++++++++
2 files changed, 22 insertions(+)
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
new file mode 100644
index 0000000000..6c834be897
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+ install -d ${D}${bindir}
+ echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+ chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 315d1a61c2..9457f1e3ac 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -35,3 +35,13 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
TESTSTRING:pn-sysroot-test-arch2 = "%s"
""" % (uuid1, uuid2))
bitbake("sysroot-test")
+
+ def test_sysroot_max_shebang(self):
+ """
+ Summary: Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+ Expected: Fail when a shebang bigger than the max shebang-size is reached.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+ res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v5 5/7] oeqa/selftest: Test staged .la and .pc files
2022-06-14 15:10 ` [PATCH v5 1/7] python: Avoid shebang overflow on python-config.py Paulo Neves
` (2 preceding siblings ...)
2022-06-14 15:11 ` [PATCH v5 4/7] oeqa/selftest: Add test for shebang overflow Paulo Neves
@ 2022-06-14 15:11 ` Paulo Neves
2022-06-14 15:11 ` [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util Paulo Neves
2022-06-14 15:11 ` [PATCH v5 7/7] libcheck: Fix too long shebang for native case Paulo Neves
5 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-14 15:11 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
These files are checked by qa_check_staged but there was no
test cases for whether the tests actually worked. Now there
are.
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
.../sysroot-test/sysroot-la-test_1.0.bb | 16 ++++++++++
.../sysroot-test/sysroot-pc-test_1.0.bb | 12 +++++++
meta/lib/oeqa/selftest/cases/sysroot.py | 32 +++++++++++++++++++
3 files changed, 60 insertions(+)
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
new file mode 100644
index 0000000000..21f06782fb
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+ install -d ${D}${libdir}/test/
+ echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
new file mode 100644
index 0000000000..e748310fc4
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+ install -d ${D}${libdir}/test/
+ echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 9457f1e3ac..588fc8c713 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -45,3 +45,35 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
self.assertTrue(expected in res.output, msg=res.output)
+
+ def test_sysroot_la(self):
+ """
+ Summary: Check that workdir paths are not contained in .la files.
+ Expected: Fail when a workdir path is found in the file content.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "la-test.la failed sanity test (workdir) in path"
+
+ res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[la]' in res.output, msg=res.output)
+
+ res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[la]' in res.output, msg=res.output)
+
+ def test_sysroot_pkgconfig(self):
+ """
+ Summary: Check that tmpdir paths are not contained in .pc files.
+ Expected: Fail when a tmpdir path is found in the file content.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "test.pc failed sanity test (tmpdir) in path"
+
+ res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+ self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+ self.assertTrue(expected in res.output, msg=res.output)
+
+ res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.
2022-06-14 15:10 ` [PATCH v5 1/7] python: Avoid shebang overflow on python-config.py Paulo Neves
` (3 preceding siblings ...)
2022-06-14 15:11 ` [PATCH v5 5/7] oeqa/selftest: Test staged .la and .pc files Paulo Neves
@ 2022-06-14 15:11 ` Paulo Neves
2022-06-15 10:52 ` [OE-core] " Ross Burton
2022-06-14 15:11 ` [PATCH v5 7/7] libcheck: Fix too long shebang for native case Paulo Neves
5 siblings, 1 reply; 64+ messages in thread
From: Paulo Neves @ 2022-06-14 15:11 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
Useful to work around shebang relocation issues, where
shebangs are too long or have arguments in them, thus preventing them
from using the /usr/bin/env shebang.
---
.../wrapper/cmdline-shebang-wrapper-test.bb | 21 ++++++++++++
.../recipes-test/wrapper/files/test.awk | 2 ++
meta/classes/utils.bbclass | 34 +++++++++++++++++++
meta/lib/oeqa/selftest/cases/wrapper.py | 11 ++++++
4 files changed, 68 insertions(+)
create mode 100644 meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
diff --git a/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
new file mode 100644
index 0000000000..c4126a41fc
--- /dev/null
+++ b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
@@ -0,0 +1,21 @@
+SUMMARY = "Check that create_cmdline_shebang works"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI += "file://test.awk"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+ install -d ${D}${bindir}
+ install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
+ sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
+ create_cmdline_shebang_wrapper ${D}${bindir}/test
+ if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
+ bbfatal "Wrapper is broken"
+ else
+ bbnote "Wrapper is good"
+ fi
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk b/meta-selftest/recipes-test/wrapper/files/test.awk
new file mode 100644
index 0000000000..91429197b1
--- /dev/null
+++ b/meta-selftest/recipes-test/wrapper/files/test.awk
@@ -0,0 +1,2 @@
+#! @AWK_BIN@ -f
+BEGIN { print "Don't Panic!" }
diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
index b4eb3d38ab..b617632d9f 100644
--- a/meta/classes/utils.bbclass
+++ b/meta/classes/utils.bbclass
@@ -184,6 +184,40 @@ END
chmod +x $cmd
}
+create_cmdline_shebang_wrapper () {
+ # Create a wrapper script where commandline options are needed
+ #
+ # These are useful to work around shebang relocation issues, where shebangs are too
+ # long or have arguments in them, thus preventing them from using the /usr/bin/env
+ # shebang
+ #
+ # Usage: create_cmdline_wrapper FILENAME <extra-options>
+
+ cmd=$1
+ shift
+
+ echo "Generating wrapper script for $cmd"
+
+ # Strip #! and get remaining interpreter + arg
+ argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )")"
+ # strip the shebang from the real script as we do not want it to be usable anyway
+ tail -n +2 $cmd > $cmd.real
+ cmdname=$(basename $cmd)
+ dirname=$(dirname $cmd)
+ cmdoptions=$@
+ if [ "${base_prefix}" != "" ]; then
+ relpath=`python3 -c "import os; print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
+ cmdoptions=`echo $@ | sed -e "s:${base_prefix}:\\$realdir/$relpath:g"`
+ fi
+ cat <<END >$cmd
+#!/usr/bin/env bash
+realpath=\`readlink -fn \$0\`
+realdir=\`dirname \$realpath\`
+exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions "\$@"
+END
+ chmod +x $cmd
+}
+
create_wrapper () {
# Create a wrapper script where extra environment variables are needed
#
diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py b/meta/lib/oeqa/selftest/cases/wrapper.py
new file mode 100644
index 0000000000..6de63310c0
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/wrapper.py
@@ -0,0 +1,11 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake
+
+class WrapperTests(OESelftestTestCase):
+ def test_shebang_wrapper(self):
+ """
+ Summary: Build a recipe which will fail if the cmdline_shebang_wrapper function is defective.
+ Expected: Exit status to be 0.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ res = bitbake("cmdline-shebang-wrapper-test -c install", ignore_status=False)
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.
2022-06-14 15:11 ` [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util Paulo Neves
@ 2022-06-15 10:52 ` Ross Burton
2022-06-15 13:11 ` Peter Kjellerstedt
0 siblings, 1 reply; 64+ messages in thread
From: Ross Burton @ 2022-06-15 10:52 UTC (permalink / raw)
To: ptsneves@gmail.com; +Cc: openembedded-core@lists.openembedded.org
On 14 Jun 2022, at 16:11, Paulo Neves via lists.openembedded.org <ptsneves=gmail.com@lists.openembedded.org> wrote:
> +create_cmdline_shebang_wrapper () {
The indentation in this function is a mix of tabs and spaces, please unify.
> + argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )”)”
[ ]* seems overly, complex, no need for the brackets. I’d anchor to the beginning of the line, and remove the g as there’s only one expected.
Might also be wise to assert that you found something here, so running this on an ELF fails with an obvious error.
Ross
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
^ permalink raw reply [flat|nested] 64+ messages in thread* RE: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.
2022-06-15 10:52 ` [OE-core] " Ross Burton
@ 2022-06-15 13:11 ` Peter Kjellerstedt
2022-06-15 13:50 ` Paulo Neves
0 siblings, 1 reply; 64+ messages in thread
From: Peter Kjellerstedt @ 2022-06-15 13:11 UTC (permalink / raw)
To: Ross Burton, ptsneves@gmail.com; +Cc: openembedded-core@lists.openembedded.org
> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Ross Burton
> Sent: den 15 juni 2022 12:52
> To: ptsneves@gmail.com
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper
> util.
>
> On 14 Jun 2022, at 16:11, Paulo Neves via lists.openembedded.org
> <ptsneves=gmail.com@lists.openembedded.org> wrote:
> > +create_cmdline_shebang_wrapper () {
>
> The indentation in this function is a mix of tabs and spaces, please
> unify.
>
> > + argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )”)”
>
> [ ]* seems overly, complex, no need for the brackets. I’d anchor to the
> beginning of the line, and remove the g as there’s only one expected.
This should work:
argument=$(basename "$(sed -ne 's/^#! *//p;q')" $cmd)
However, if the first line is something like "#!/usr/bin/env python3", then
$argument will be "env python3" and somehow I assume that is not exactly
what the rest of the code expects.
>
> Might also be wise to assert that you found something here, so running
> this on an ELF fails with an obvious error.
>
> Ross
//Peter
^ permalink raw reply [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.
2022-06-15 13:11 ` Peter Kjellerstedt
@ 2022-06-15 13:50 ` Paulo Neves
2022-06-15 22:16 ` Peter Kjellerstedt
0 siblings, 1 reply; 64+ messages in thread
From: Paulo Neves @ 2022-06-15 13:50 UTC (permalink / raw)
To: Peter Kjellerstedt, Ross Burton; +Cc: openembedded-core@lists.openembedded.org
On 6/15/22 15:11, Peter Kjellerstedt wrote:
>> -----Original Message-----
>> From: openembedded-core@lists.openembedded.org <openembedded-
>> core@lists.openembedded.org> On Behalf Of Ross Burton
>> Sent: den 15 juni 2022 12:52
>> To: ptsneves@gmail.com
>> Cc: openembedded-core@lists.openembedded.org
>> Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper
>> util.
>>
>> On 14 Jun 2022, at 16:11, Paulo Neves via lists.openembedded.org
>> <ptsneves=gmail.com@lists.openembedded.org> wrote:
>>> +create_cmdline_shebang_wrapper () {
>> The indentation in this function is a mix of tabs and spaces, please
>> unify.
>>
>>> + argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )”)”
>> [ ]* seems overly, complex, no need for the brackets. I’d anchor to the
>> beginning of the line, and remove the g as there’s only one expected.
> This should work:
>
> argument=$(basename "$(sed -ne 's/^#! *//p;q')" $cmd)
>
> However, if the first line is something like "#!/usr/bin/env python3", then
> $argument will be "env python3" and somehow I assume that is not exactly
> what the rest of the code expects.
The use case is that whatever is in shebang should be extracted and ran
verbatim, so truncating like your suggestion can work but has slightly
different semantic.
>> Might also be wise to assert that you found something here, so running
>> this on an ELF fails with an obvious error.
>>
>> Ross
> //Peter
>
Paulo Neves
^ permalink raw reply [flat|nested] 64+ messages in thread* RE: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.
2022-06-15 13:50 ` Paulo Neves
@ 2022-06-15 22:16 ` Peter Kjellerstedt
2022-06-19 19:03 ` Paulo Neves
0 siblings, 1 reply; 64+ messages in thread
From: Peter Kjellerstedt @ 2022-06-15 22:16 UTC (permalink / raw)
To: Paulo Neves, Ross Burton; +Cc: openembedded-core@lists.openembedded.org
> -----Original Message-----
> From: Paulo Neves <ptsneves@gmail.com>
> Sent: den 15 juni 2022 15:51
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; Ross Burton
> <ross.burton@arm.com>
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper
> util.
>
> On 6/15/22 15:11, Peter Kjellerstedt wrote:
> >> -----Original Message-----
> >> From: openembedded-core@lists.openembedded.org <openembedded-
> >> core@lists.openembedded.org> On Behalf Of Ross Burton
> >> Sent: den 15 juni 2022 12:52
> >> To: ptsneves@gmail.com
> >> Cc: openembedded-core@lists.openembedded.org
> >> Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add
> cmdline_shebang_wrapper
> >> util.
> >>
> >> On 14 Jun 2022, at 16:11, Paulo Neves via lists.openembedded.org
> >> <ptsneves=gmail.com@lists.openembedded.org> wrote:
> >>> +create_cmdline_shebang_wrapper () {
> >> The indentation in this function is a mix of tabs and spaces, please
> >> unify.
> >>
> >>> + argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )”)”
> >> [ ]* seems overly, complex, no need for the brackets. I’d anchor to the
> >> beginning of the line, and remove the g as there’s only one expected.
> > This should work:
> >
> > argument=$(basename "$(sed -ne 's/^#! *//p;q')" $cmd)
> >
> > However, if the first line is something like "#!/usr/bin/env python3", then
> > $argument will be "env python3" and somehow I assume that is not exactly
> > what the rest of the code expects.
>
> The use case is that whatever is in shebang should be extracted and ran
> verbatim, so truncating like your suggestion can work but has slightly
> different semantic.
Umm, my suggested line above should give exactly the same output as yours,
only with fewer commands...
> >> Might also be wise to assert that you found something here, so running
> >> this on an ELF fails with an obvious error.
> >>
> >> Ross
> > //Peter
> >
> Paulo Neves
//Peter
^ permalink raw reply [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util.
2022-06-15 22:16 ` Peter Kjellerstedt
@ 2022-06-19 19:03 ` Paulo Neves
0 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-19 19:03 UTC (permalink / raw)
To: Peter Kjellerstedt, Ross Burton; +Cc: openembedded-core@lists.openembedded.org
You are right never mind.
Paulo Neves
On 6/16/22 00:16, Peter Kjellerstedt wrote:
>> -----Original Message-----
>> From: Paulo Neves <ptsneves@gmail.com>
>> Sent: den 15 juni 2022 15:51
>> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; Ross Burton
>> <ross.burton@arm.com>
>> Cc: openembedded-core@lists.openembedded.org
>> Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper
>> util.
>>
>> On 6/15/22 15:11, Peter Kjellerstedt wrote:
>>>> -----Original Message-----
>>>> From: openembedded-core@lists.openembedded.org <openembedded-
>>>> core@lists.openembedded.org> On Behalf Of Ross Burton
>>>> Sent: den 15 juni 2022 12:52
>>>> To: ptsneves@gmail.com
>>>> Cc: openembedded-core@lists.openembedded.org
>>>> Subject: Re: [OE-core] [PATCH v5 6/7] utils: Add
>> cmdline_shebang_wrapper
>>>> util.
>>>>
>>>> On 14 Jun 2022, at 16:11, Paulo Neves via lists.openembedded.org
>>>> <ptsneves=gmail.com@lists.openembedded.org> wrote:
>>>>> +create_cmdline_shebang_wrapper () {
>>>> The indentation in this function is a mix of tabs and spaces, please
>>>> unify.
>>>>
>>>>> + argument="$(basename "$(head -n1 $cmd | sed -e 's|#![ ]*||g' )”)”
>>>> [ ]* seems overly, complex, no need for the brackets. I’d anchor to the
>>>> beginning of the line, and remove the g as there’s only one expected.
>>> This should work:
>>>
>>> argument=$(basename "$(sed -ne 's/^#! *//p;q')" $cmd)
>>>
>>> However, if the first line is something like "#!/usr/bin/env python3", then
>>> $argument will be "env python3" and somehow I assume that is not exactly
>>> what the rest of the code expects.
>> The use case is that whatever is in shebang should be extracted and ran
>> verbatim, so truncating like your suggestion can work but has slightly
>> different semantic.
> Umm, my suggested line above should give exactly the same output as yours,
> only with fewer commands...
>
>>>> Might also be wise to assert that you found something here, so running
>>>> this on an ELF fails with an obvious error.
>>>>
>>>> Ross
>>> //Peter
>>>
>> Paulo Neves
> //Peter
>
^ permalink raw reply [flat|nested] 64+ messages in thread
* [PATCH v5 7/7] libcheck: Fix too long shebang for native case.
2022-06-14 15:10 ` [PATCH v5 1/7] python: Avoid shebang overflow on python-config.py Paulo Neves
` (4 preceding siblings ...)
2022-06-14 15:11 ` [PATCH v5 6/7] utils: Add cmdline_shebang_wrapper util Paulo Neves
@ 2022-06-14 15:11 ` Paulo Neves
5 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-14 15:11 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
It requires a shebang wrapper due to the fact that awk interpreter
has an argument.
---
meta/recipes-support/libcheck/libcheck_0.15.2.bb | 3 +++
1 file changed, 3 insertions(+)
diff --git a/meta/recipes-support/libcheck/libcheck_0.15.2.bb b/meta/recipes-support/libcheck/libcheck_0.15.2.bb
index 188d689cc3..1393aa2a1c 100644
--- a/meta/recipes-support/libcheck/libcheck_0.15.2.bb
+++ b/meta/recipes-support/libcheck/libcheck_0.15.2.bb
@@ -23,6 +23,9 @@ CACHED_CONFIGUREVARS += "ac_cv_path_AWK_PATH=${bindir}/gawk"
RREPLACES:${PN} = "check (<= 0.9.5)"
+do_install:append:class-native() {
+ create_cmdline_shebang_wrapper ${D}${bindir}/checkmk
+}
BBCLASSEXTEND = "native nativesdk"
PACKAGES =+ "checkmk"
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread
* Re: [OE-core] [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs
2022-06-06 15:58 [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
` (5 preceding siblings ...)
2022-06-14 15:10 ` [PATCH v5 1/7] python: Avoid shebang overflow on python-config.py Paulo Neves
@ 2022-06-15 11:07 ` Ross Burton
2022-06-15 11:08 ` Ross Burton
2022-06-19 19:20 ` [PATCH v6 1/7] python: Avoid shebang overflow on python-config.py Paulo Neves
7 siblings, 1 reply; 64+ messages in thread
From: Ross Burton @ 2022-06-15 11:07 UTC (permalink / raw)
To: ptsneves@gmail.com; +Cc: openembedded-core@lists.openembedded.org
> + if not skip_shebang_size:
> + errors = {}
> + package_qa_check_shebang_size(path, "", d, None, errors)
> + for e in errors:
> + oe.qa.handle_error(e, errors[e], d)
Where is package_qa_check_shebang_size() defined?
Ross
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
^ permalink raw reply [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs
2022-06-15 11:07 ` [OE-core] [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs Ross Burton
@ 2022-06-15 11:08 ` Ross Burton
2022-06-15 11:09 ` Paulo Neves
0 siblings, 1 reply; 64+ messages in thread
From: Ross Burton @ 2022-06-15 11:08 UTC (permalink / raw)
To: ptsneves@gmail.com; +Cc: openembedded-core@lists.openembedded.org
Never mind, can’t read :)
> On 15 Jun 2022, at 12:07, Ross Burton <Ross.Burton@arm.com> wrote:
>
>> + if not skip_shebang_size:
>> + errors = {}
>> + package_qa_check_shebang_size(path, "", d, None, errors)
>> + for e in errors:
>> + oe.qa.handle_error(e, errors[e], d)
>
> Where is package_qa_check_shebang_size() defined?
>
> Ross
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
^ permalink raw reply [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs
2022-06-15 11:08 ` Ross Burton
@ 2022-06-15 11:09 ` Paulo Neves
0 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-15 11:09 UTC (permalink / raw)
To: Ross Burton; +Cc: openembedded-core@lists.openembedded.org
:)
The function should probably be refactored so it does not imply it is
only useful for packaging.
Paulo Neves
On 6/15/22 13:08, Ross Burton wrote:
> Never mind, can’t read :)
>
>> On 15 Jun 2022, at 12:07, Ross Burton <Ross.Burton@arm.com> wrote:
>>
>>> + if not skip_shebang_size:
>>> + errors = {}
>>> + package_qa_check_shebang_size(path, "", d, None, errors)
>>> + for e in errors:
>>> + oe.qa.handle_error(e, errors[e], d)
>> Where is package_qa_check_shebang_size() defined?
>>
>> Ross
> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
^ permalink raw reply [flat|nested] 64+ messages in thread
* [PATCH v6 1/7] python: Avoid shebang overflow on python-config.py
2022-06-06 15:58 [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
` (6 preceding siblings ...)
2022-06-15 11:07 ` [OE-core] [PATCH 1/3] insane.bbclass: Make do_qa_staging check shebangs Ross Burton
@ 2022-06-19 19:20 ` Paulo Neves
2022-06-19 19:20 ` [PATCH v6 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2 Paulo Neves
` (5 more replies)
7 siblings, 6 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-19 19:20 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
The native path may be too big, leading to shebang
overflow. Just use the #!/usr/bin/env python3.
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
...shebang-overflow-on-python-config.py.patch | 33 +++++++++++++++++++
.../recipes-devtools/python/python3_3.10.4.bb | 1 +
2 files changed, 34 insertions(+)
create mode 100644 meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
diff --git a/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
new file mode 100644
index 0000000000..921da8de7c
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/0001-Avoid-shebang-overflow-on-python-config.py.patch
@@ -0,0 +1,33 @@
+From f0c9dec63d452a7cd1e15ea653f4aced281f021c Mon Sep 17 00:00:00 2001
+From: Paulo Neves <ptsneves@gmail.com>
+Date: Tue, 7 Jun 2022 16:16:41 +0200
+Subject: [PATCH 1/1] Avoid shebang overflow on python-config.py
+
+The whole native path may be too big, leading to shebang
+overflow. Let's just use the env shebang.
+
+Denial reason: [1]
+
+Upstream-Status: Denied [distribution]
+
+[1] https://github.com/python/cpython/pull/93760#pullrequestreview-1005365737
+---
+ Makefile.pre.in | 2 ++
+ 1 file changed, 2 insertions(+)
+
+diff --git a/Makefile.pre.in b/Makefile.pre.in
+index f0aedb76cb58999427804255da56fa53284d7032..dd88e43114730f7681715777cc76dabb31113176 100644
+--- a/Makefile.pre.in
++++ b/Makefile.pre.in
+@@ -1638,6 +1638,8 @@ python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
+ @ # Substitution happens here, as the completely-expanded BINDIR
+ @ # is not available in configure
+ sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
++ @ # Otherwise we might get huge shebangs with native paths
++ sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' python-config.py
+ @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
+ LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
+ @ # In OpenEmbedded, always use the python version of the script, the shell
+--
+2.25.1
+
diff --git a/meta/recipes-devtools/python/python3_3.10.4.bb b/meta/recipes-devtools/python/python3_3.10.4.bb
index 6bd3a6aba8..357025f856 100644
--- a/meta/recipes-devtools/python/python3_3.10.4.bb
+++ b/meta/recipes-devtools/python/python3_3.10.4.bb
@@ -34,6 +34,7 @@ SRC_URI = "http://www.python.org/ftp/python/${PV}/Python-${PV}.tar.xz \
file://0001-Lib-pty.py-handle-stdin-I-O-errors-same-way-as-maste.patch \
file://0001-setup.py-Do-not-detect-multiarch-paths-when-cross-co.patch \
file://deterministic_imports.patch \
+ file://0001-Avoid-shebang-overflow-on-python-config.py.patch \
"
SRC_URI:append:class-native = " \
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v6 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2
2022-06-19 19:20 ` [PATCH v6 1/7] python: Avoid shebang overflow on python-config.py Paulo Neves
@ 2022-06-19 19:20 ` Paulo Neves
2022-06-19 19:20 ` [PATCH v6 3/7] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
` (4 subsequent siblings)
5 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-19 19:20 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
A native path can cause a shebang overflow on gtkdoc-mkhtml.
Replace it with /usr/bin/env.
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb | 1 +
1 file changed, 1 insertion(+)
diff --git a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
index 150d2c0b23..392913fcc6 100644
--- a/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
+++ b/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb
@@ -43,6 +43,7 @@ do_install:append () {
${datadir}/gtk-doc/python/gtkdoc/config.py; do
sed -e 's,${RECIPE_SYSROOT_NATIVE}/usr/bin/pkg-config,${bindir}/pkg-config,' \
-e 's,${HOSTTOOLS_DIR}/python3,${bindir}/python3,' \
+ -e '1s|^#!.*|#!/usr/bin/env python3|' \
-i ${D}$fn
done
}
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v6 3/7] insane.bbclass: Make do_qa_staging check shebangs
2022-06-19 19:20 ` [PATCH v6 1/7] python: Avoid shebang overflow on python-config.py Paulo Neves
2022-06-19 19:20 ` [PATCH v6 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2 Paulo Neves
@ 2022-06-19 19:20 ` Paulo Neves
2022-06-19 19:20 ` [PATCH v6 4/7] oeqa/selftest: Add test for shebang overflow Paulo Neves
` (3 subsequent siblings)
5 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-19 19:20 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
As reported in the bug report [1], there was no check for shebang
sizes on native scripts and now this is fixed.
The path scope of the qa_staging was increased from just checking
libdir to all the relevant SYSROOT_DIRS.
It is possible to skip this check through INSANE_SKIP.
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
meta/classes/insane.bbclass | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 9ca84bace9..b2951a48fe 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -630,6 +630,11 @@ def qa_check_staged(path,d):
bb.note("Recipe %s skipping qa checking: pkgconfig" % d.getVar('PN'))
skip_pkgconfig = True
+ skip_shebang_size = False
+ if 'shebang-size' in skip:
+ bb.note("Recipe %s skipping qa checkking: shebang-size" % d.getVar('PN'))
+ skip_shebang_size = True
+
# find all .la and .pc files
# read the content
# and check for stuff that looks wrong
@@ -651,6 +656,13 @@ def qa_check_staged(path,d):
error_msg = "%s failed sanity test (tmpdir) in path %s" % (file,root)
oe.qa.handle_error("pkgconfig", error_msg, d)
+ if not skip_shebang_size:
+ errors = {}
+ package_qa_check_shebang_size(path, "", d, None, errors)
+ for e in errors:
+ oe.qa.handle_error(e, errors[e], d)
+
+
# Run all package-wide warnfuncs and errorfuncs
def package_qa_package(warnfuncs, errorfuncs, package, d):
warnings = {}
@@ -1139,7 +1151,9 @@ addtask do_package_qa_setscene
python do_qa_staging() {
bb.note("QA checking staging")
- qa_check_staged(d.expand('${SYSROOT_DESTDIR}${libdir}'), d)
+ sysroot_destdir = d.expand('${SYSROOT_DESTDIR}')
+ for sysroot_dir in d.expand('${SYSROOT_DIRS}').split():
+ qa_check_staged(sysroot_destdir + sysroot_dir, d)
oe.qa.exit_with_message_if_errors("QA staging was broken by the package built above", d)
}
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v6 4/7] oeqa/selftest: Add test for shebang overflow
2022-06-19 19:20 ` [PATCH v6 1/7] python: Avoid shebang overflow on python-config.py Paulo Neves
2022-06-19 19:20 ` [PATCH v6 2/7] gtk-doc: Fix potential shebang overflow on gtkdoc-mkhtml2 Paulo Neves
2022-06-19 19:20 ` [PATCH v6 3/7] insane.bbclass: Make do_qa_staging check shebangs Paulo Neves
@ 2022-06-19 19:20 ` Paulo Neves
2022-06-19 19:20 ` [PATCH v6 5/7] oeqa/selftest: Test staged .la and .pc files Paulo Neves
` (2 subsequent siblings)
5 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-19 19:20 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
Make sure we do not stage any executable with a bigger shebang
than 128. Fixes [1]
[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=11053
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
.../sysroot-test/sysroot-shebang-test_1.0.bb | 12 ++++++++++++
meta/lib/oeqa/selftest/cases/sysroot.py | 11 +++++++++++
2 files changed, 23 insertions(+)
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
new file mode 100644
index 0000000000..6c834be897
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-shebang-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Check that shebang does not exceed 128 characters"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+ install -d ${D}${bindir}
+ echo '#!BiM3cnVd1Amtv6PG+FynrQiVMbZnX5ELgF21q3EkuB+44JEGWtq8TvBJ7EGidfVs3eR3wVOUbLnjYDlKUWcm7YC/ute7f+KDHbwxziRUSUBZAUqgjiQdfQ0HnxajI0ozbM863E9JV9k13yZKYfh9/zR77Y6Dl4Dd3zOWS75LSpkAXV' > ${D}${bindir}/max-shebang
+ chmod 755 ${D}${bindir}/max-shebang
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index 315d1a61c2..a17fd1818a 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -35,3 +35,14 @@ TESTSTRING:pn-sysroot-test-arch1 = "%s"
TESTSTRING:pn-sysroot-test-arch2 = "%s"
""" % (uuid1, uuid2))
bitbake("sysroot-test")
+
+ def test_sysroot_max_shebang(self):
+ """
+ Summary: Check max shebang triggers. To confirm [YOCTO #11053] is closed.
+ Expected: Fail when a shebang bigger than the max shebang-size is reached.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "maximum shebang size exceeded, the maximum size is 128. [shebang-size]"
+ res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue(res.status != 0)
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v6 5/7] oeqa/selftest: Test staged .la and .pc files
2022-06-19 19:20 ` [PATCH v6 1/7] python: Avoid shebang overflow on python-config.py Paulo Neves
` (2 preceding siblings ...)
2022-06-19 19:20 ` [PATCH v6 4/7] oeqa/selftest: Add test for shebang overflow Paulo Neves
@ 2022-06-19 19:20 ` Paulo Neves
2022-06-19 19:20 ` [PATCH v6 6/7] utils: Add cmdline_shebang_wrapper util Paulo Neves
2022-06-19 19:20 ` [PATCH v6 7/7] libcheck: Fix too long shebang for native case Paulo Neves
5 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-19 19:20 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
These files are checked by qa_check_staged but there was no
test cases for whether the tests actually worked. Now there
are.
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
.../sysroot-test/sysroot-la-test_1.0.bb | 16 +++++++++
.../sysroot-test/sysroot-pc-test_1.0.bb | 12 +++++++
meta/lib/oeqa/selftest/cases/sysroot.py | 36 +++++++++++++++++++
3 files changed, 64 insertions(+)
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
create mode 100644 meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
new file mode 100644
index 0000000000..21f06782fb
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-la-test_1.0.bb
@@ -0,0 +1,16 @@
+SUMMARY = "Produce a broken la file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+# remove-libtool.bbclass is inherited by default and removes all
+# .la files which for this test we specifically do not want.
+REMOVE_LIBTOOL_LA = "0"
+
+do_install() {
+ install -d ${D}${libdir}/test/
+ echo '${WORKDIR}' > ${D}${libdir}/test/la-test.la
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
new file mode 100644
index 0000000000..e748310fc4
--- /dev/null
+++ b/meta-selftest/recipes-test/sysroot-test/sysroot-pc-test_1.0.bb
@@ -0,0 +1,12 @@
+SUMMARY = "Produce a broken pc file"
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+EXCLUDE_FROM_WORLD = "1"
+
+do_install() {
+ install -d ${D}${libdir}/test/
+ echo '${WORKDIR}' > ${D}${libdir}/test/test.pc
+}
+
+BBCLASSEXTEND += "native"
diff --git a/meta/lib/oeqa/selftest/cases/sysroot.py b/meta/lib/oeqa/selftest/cases/sysroot.py
index a17fd1818a..294ba4a4a0 100644
--- a/meta/lib/oeqa/selftest/cases/sysroot.py
+++ b/meta/lib/oeqa/selftest/cases/sysroot.py
@@ -46,3 +46,39 @@ TESTSTRING:pn-sysroot-test-arch2 = "%s"
res = bitbake("sysroot-shebang-test-native -c populate_sysroot", ignore_status=True)
self.assertTrue(expected in res.output, msg=res.output)
self.assertTrue(res.status != 0)
+
+ def test_sysroot_la(self):
+ """
+ Summary: Check that workdir paths are not contained in .la files.
+ Expected: Fail when a workdir path is found in the file content.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "la-test.la failed sanity test (workdir) in path"
+
+ res = bitbake("sysroot-la-test -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[la]' in res.output, msg=res.output)
+ self.assertTrue(res.status != 0)
+
+ res = bitbake("sysroot-la-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[la]' in res.output, msg=res.output)
+ self.assertTrue(res.status != 0)
+
+ def test_sysroot_pkgconfig(self):
+ """
+ Summary: Check that tmpdir paths are not contained in .pc files.
+ Expected: Fail when a tmpdir path is found in the file content.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ expected = "test.pc failed sanity test (tmpdir) in path"
+
+ res = bitbake("sysroot-pc-test -c populate_sysroot", ignore_status=True)
+ self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue(res.status != 0)
+
+ res = bitbake("sysroot-pc-test-native -c populate_sysroot", ignore_status=True)
+ self.assertTrue(expected in res.output, msg=res.output)
+ self.assertTrue('[pkgconfig]' in res.output, msg=res.output)
+ self.assertTrue(res.status != 0)
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* [PATCH v6 6/7] utils: Add cmdline_shebang_wrapper util.
2022-06-19 19:20 ` [PATCH v6 1/7] python: Avoid shebang overflow on python-config.py Paulo Neves
` (3 preceding siblings ...)
2022-06-19 19:20 ` [PATCH v6 5/7] oeqa/selftest: Test staged .la and .pc files Paulo Neves
@ 2022-06-19 19:20 ` Paulo Neves
2022-07-05 4:50 ` [OE-core] " Jacob Kroon
2022-06-19 19:20 ` [PATCH v6 7/7] libcheck: Fix too long shebang for native case Paulo Neves
5 siblings, 1 reply; 64+ messages in thread
From: Paulo Neves @ 2022-06-19 19:20 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
Useful to work around shebang relocation issues, where
shebangs are too long or have arguments in them, thus preventing them
from using the /usr/bin/env shebang.
---
.../wrapper/cmdline-shebang-wrapper-test.bb | 21 ++++++++++++
.../recipes-test/wrapper/files/test.awk | 2 ++
meta/classes/utils.bbclass | 34 +++++++++++++++++++
meta/lib/oeqa/selftest/cases/wrapper.py | 11 ++++++
4 files changed, 68 insertions(+)
create mode 100644 meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
diff --git a/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
new file mode 100644
index 0000000000..c4126a41fc
--- /dev/null
+++ b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
@@ -0,0 +1,21 @@
+SUMMARY = "Check that create_cmdline_shebang works"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI += "file://test.awk"
+
+EXCLUDE_FROM_WORLD = "1"
+do_install() {
+ install -d ${D}${bindir}
+ install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
+ sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
+ create_cmdline_shebang_wrapper ${D}${bindir}/test
+ if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
+ bbfatal "Wrapper is broken"
+ else
+ bbnote "Wrapper is good"
+ fi
+}
+
+BBCLASSEXTEND = "native"
diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk b/meta-selftest/recipes-test/wrapper/files/test.awk
new file mode 100644
index 0000000000..91429197b1
--- /dev/null
+++ b/meta-selftest/recipes-test/wrapper/files/test.awk
@@ -0,0 +1,2 @@
+#! @AWK_BIN@ -f
+BEGIN { print "Don't Panic!" }
diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
index b4eb3d38ab..b58c22771f 100644
--- a/meta/classes/utils.bbclass
+++ b/meta/classes/utils.bbclass
@@ -184,6 +184,40 @@ END
chmod +x $cmd
}
+create_cmdline_shebang_wrapper () {
+ # Create a wrapper script where commandline options are needed
+ #
+ # These are useful to work around shebang relocation issues, where shebangs are too
+ # long or have arguments in them, thus preventing them from using the /usr/bin/env
+ # shebang
+ #
+ # Usage: create_cmdline_wrapper FILENAME <extra-options>
+
+ cmd=$1
+ shift
+
+ echo "Generating wrapper script for $cmd"
+
+ # Strip #! and get remaining interpreter + arg
+ argument="$(sed -ne 's/^#! *//p;q' $cmd)"
+ # strip the shebang from the real script as we do not want it to be usable anyway
+ tail -n +2 $cmd > $cmd.real
+ cmdname=$(basename $cmd)
+ dirname=$(dirname $cmd)
+ cmdoptions=$@
+ if [ "${base_prefix}" != "" ]; then
+ relpath=`python3 -c "import os; print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
+ cmdoptions=`echo $@ | sed -e "s:${base_prefix}:\\$realdir/$relpath:g"`
+ fi
+ cat <<END >$cmd
+#!/usr/bin/env bash
+realpath=\`readlink -fn \$0\`
+realdir=\`dirname \$realpath\`
+exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions "\$@"
+END
+ chmod +x $cmd
+}
+
create_wrapper () {
# Create a wrapper script where extra environment variables are needed
#
diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py b/meta/lib/oeqa/selftest/cases/wrapper.py
new file mode 100644
index 0000000000..6de63310c0
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/wrapper.py
@@ -0,0 +1,11 @@
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake
+
+class WrapperTests(OESelftestTestCase):
+ def test_shebang_wrapper(self):
+ """
+ Summary: Build a recipe which will fail if the cmdline_shebang_wrapper function is defective.
+ Expected: Exit status to be 0.
+ Author: Paulo Neves <ptsneves@gmail.com>
+ """
+ res = bitbake("cmdline-shebang-wrapper-test -c install", ignore_status=False)
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread* Re: [OE-core] [PATCH v6 6/7] utils: Add cmdline_shebang_wrapper util.
2022-06-19 19:20 ` [PATCH v6 6/7] utils: Add cmdline_shebang_wrapper util Paulo Neves
@ 2022-07-05 4:50 ` Jacob Kroon
0 siblings, 0 replies; 64+ messages in thread
From: Jacob Kroon @ 2022-07-05 4:50 UTC (permalink / raw)
To: Paulo Neves, openembedded-core
On 6/19/22 21:20, Paulo Neves wrote:
> Useful to work around shebang relocation issues, where
> shebangs are too long or have arguments in them, thus preventing them
> from using the /usr/bin/env shebang.
> ---
> .../wrapper/cmdline-shebang-wrapper-test.bb | 21 ++++++++++++
> .../recipes-test/wrapper/files/test.awk | 2 ++
> meta/classes/utils.bbclass | 34 +++++++++++++++++++
> meta/lib/oeqa/selftest/cases/wrapper.py | 11 ++++++
> 4 files changed, 68 insertions(+)
> create mode 100644 meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
> create mode 100644 meta-selftest/recipes-test/wrapper/files/test.awk
> create mode 100644 meta/lib/oeqa/selftest/cases/wrapper.py
>
> diff --git a/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
> new file mode 100644
> index 0000000000..c4126a41fc
> --- /dev/null
> +++ b/meta-selftest/recipes-test/wrapper/cmdline-shebang-wrapper-test.bb
> @@ -0,0 +1,21 @@
> +SUMMARY = "Check that create_cmdline_shebang works"
> +LICENSE = "MIT"
> +LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
> +INHIBIT_DEFAULT_DEPS = "1"
> +
> +SRC_URI += "file://test.awk"
> +
> +EXCLUDE_FROM_WORLD = "1"
> +do_install() {
> + install -d ${D}${bindir}
> + install -m 0755 ${WORKDIR}/test.awk ${D}${bindir}/test
> + sed -i -e 's|@AWK_BIN@|${bindir}/awk|g' ${D}${bindir}/test
> + create_cmdline_shebang_wrapper ${D}${bindir}/test
> + if [ $(${D}${bindir}/test) != "Don't Panic!" ]; then
> + bbfatal "Wrapper is broken"
> + else
> + bbnote "Wrapper is good"
> + fi
> +}
> +
> +BBCLASSEXTEND = "native"
> diff --git a/meta-selftest/recipes-test/wrapper/files/test.awk b/meta-selftest/recipes-test/wrapper/files/test.awk
> new file mode 100644
> index 0000000000..91429197b1
> --- /dev/null
> +++ b/meta-selftest/recipes-test/wrapper/files/test.awk
> @@ -0,0 +1,2 @@
> +#! @AWK_BIN@ -f
> +BEGIN { print "Don't Panic!" }
> diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
> index b4eb3d38ab..b58c22771f 100644
> --- a/meta/classes/utils.bbclass
> +++ b/meta/classes/utils.bbclass
> @@ -184,6 +184,40 @@ END
> chmod +x $cmd
> }
>
> +create_cmdline_shebang_wrapper () {
> + # Create a wrapper script where commandline options are needed
> + #
> + # These are useful to work around shebang relocation issues, where shebangs are too
> + # long or have arguments in them, thus preventing them from using the /usr/bin/env
> + # shebang
> + #
> + # Usage: create_cmdline_wrapper FILENAME <extra-options>
> +
> + cmd=$1
> + shift
> +
> + echo "Generating wrapper script for $cmd"
> +
> + # Strip #! and get remaining interpreter + arg
> + argument="$(sed -ne 's/^#! *//p;q' $cmd)"
> + # strip the shebang from the real script as we do not want it to be usable anyway
> + tail -n +2 $cmd > $cmd.real
> + cmdname=$(basename $cmd)
> + dirname=$(dirname $cmd)
> + cmdoptions=$@
> + if [ "${base_prefix}" != "" ]; then
> + relpath=`python3 -c "import os; print(os.path.relpath('${D}${base_prefix}', '$dirname'))"`
> + cmdoptions=`echo $@ | sed -e "s:${base_prefix}:\\$realdir/$relpath:g"`
> + fi
> + cat <<END >$cmd
> +#!/usr/bin/env bash
> +realpath=\`readlink -fn \$0\`
> +realdir=\`dirname \$realpath\`
> +exec -a \$realdir/$cmdname $argument \$realdir/$cmdname.real $cmdoptions "\$@"
> +END
> + chmod +x $cmd
> +}
> +
Maybe this has already been raised before, but the wrapper above does a
couple of addtitional forks, which could be avoided if we use a python
wrapper script instead.
Jacob
> create_wrapper () {
> # Create a wrapper script where extra environment variables are needed
> #
> diff --git a/meta/lib/oeqa/selftest/cases/wrapper.py b/meta/lib/oeqa/selftest/cases/wrapper.py
> new file mode 100644
> index 0000000000..6de63310c0
> --- /dev/null
> +++ b/meta/lib/oeqa/selftest/cases/wrapper.py
> @@ -0,0 +1,11 @@
> +from oeqa.selftest.case import OESelftestTestCase
> +from oeqa.utils.commands import bitbake
> +
> +class WrapperTests(OESelftestTestCase):
> + def test_shebang_wrapper(self):
> + """
> + Summary: Build a recipe which will fail if the cmdline_shebang_wrapper function is defective.
> + Expected: Exit status to be 0.
> + Author: Paulo Neves <ptsneves@gmail.com>
> + """
> + res = bitbake("cmdline-shebang-wrapper-test -c install", ignore_status=False)
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#167044): https://lists.openembedded.org/g/openembedded-core/message/167044
> Mute This Topic: https://lists.openembedded.org/mt/91863579/4454410
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [jacob.kroon@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
^ permalink raw reply [flat|nested] 64+ messages in thread
* [PATCH v6 7/7] libcheck: Fix too long shebang for native case.
2022-06-19 19:20 ` [PATCH v6 1/7] python: Avoid shebang overflow on python-config.py Paulo Neves
` (4 preceding siblings ...)
2022-06-19 19:20 ` [PATCH v6 6/7] utils: Add cmdline_shebang_wrapper util Paulo Neves
@ 2022-06-19 19:20 ` Paulo Neves
5 siblings, 0 replies; 64+ messages in thread
From: Paulo Neves @ 2022-06-19 19:20 UTC (permalink / raw)
To: openembedded-core; +Cc: Paulo Neves
It requires a shebang wrapper due to the fact that awk interpreter
has an argument.
---
meta/recipes-support/libcheck/libcheck_0.15.2.bb | 3 +++
1 file changed, 3 insertions(+)
diff --git a/meta/recipes-support/libcheck/libcheck_0.15.2.bb b/meta/recipes-support/libcheck/libcheck_0.15.2.bb
index 188d689cc3..1393aa2a1c 100644
--- a/meta/recipes-support/libcheck/libcheck_0.15.2.bb
+++ b/meta/recipes-support/libcheck/libcheck_0.15.2.bb
@@ -23,6 +23,9 @@ CACHED_CONFIGUREVARS += "ac_cv_path_AWK_PATH=${bindir}/gawk"
RREPLACES:${PN} = "check (<= 0.9.5)"
+do_install:append:class-native() {
+ create_cmdline_shebang_wrapper ${D}${bindir}/checkmk
+}
BBCLASSEXTEND = "native nativesdk"
PACKAGES =+ "checkmk"
--
2.25.1
^ permalink raw reply related [flat|nested] 64+ messages in thread