[PATCH] libc-test: add libc testsuite for musl

[PATCH] libc-test: add libc testsuite for musl

[Chase Qi]

libc-test is a collection of unit test to measure the correctness and
robustness of a C/POSIX standard library implementation. It is developed
as part of the musl project.

Signed-off-by: Chase Qi <chase.qi@linaro.org>
---
 meta/recipes-core/musl/libc-test/run-ptest | 17 ++++++++
 meta/recipes-core/musl/libc-test_git.bb    | 45 ++++++++++++++++++++++
 2 files changed, 62 insertions(+)
 create mode 100644 meta/recipes-core/musl/libc-test/run-ptest
 create mode 100644 meta/recipes-core/musl/libc-test_git.bb

diff --git a/meta/recipes-core/musl/libc-test/run-ptest b/meta/recipes-core/musl/libc-test/run-ptest
new file mode 100644
index 0000000000..30abcdfba5
--- /dev/null
+++ b/meta/recipes-core/musl/libc-test/run-ptest
@@ -0,0 +1,17 @@
+#!/bin/sh
+
+make cleanall
+make run | tee libc-test.log
+
+echo ""
+echo "--- ptest result ---"
+if grep -q '^FAIL src.*\.exe.*' libc-test.log; then
+    # Consolidate output format.
+    # e.g. "FAIL src/api/main.exe [status 1]" -> "FAIL: api_main"
+    grep '^FAIL src.*\.exe.*' libc-test.log \
+        | sed 's|^FAIL src/|FAIL: |;s|/|_|;s|\.exe.*\]||'
+    exit 1
+else
+    echo "PASS: libc-test"
+    exit 0
+fi
diff --git a/meta/recipes-core/musl/libc-test_git.bb b/meta/recipes-core/musl/libc-test_git.bb
new file mode 100644
index 0000000000..d5d8845727
--- /dev/null
+++ b/meta/recipes-core/musl/libc-test_git.bb
@@ -0,0 +1,45 @@
+SUMMARY = "Musl libc test to be run with ptest"
+HOMEPAGE = "https://wiki.musl-libc.org/libc-test.html"
+DESCRIPTION = "libc-test is a collection of unit tests to measure the \
+correctness and robustness of a C/POSIX standard library implementation. It is \
+developed as part of the musl project."
+SECTION = "tests"
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://COPYRIGHT;md5=43ed1245085be90dc934288117d55a3b"
+
+inherit ptest
+
+SRCREV = "18e28496adee3d84fefdda6efcb9c5b8996a2398"
+SRC_URI = " \
+    git://nsz.repo.hu:49100/repo/libc-test;branch=master \
+    file://run-ptest \
+"
+MIRRORS += "git://nsz.repo.hu:49100/repo/libc-test git://repo.or.cz/libc-test"
+
+S = "${WORKDIR}/git"
+
+# libc-test 'make' or 'make run' command is designed to build and run tests. It
+# reports both build and test failures. The commands should be run on target.
+do_compile() {
+    :
+}
+
+RDEPENDS:${PN}-ptest = " \
+    bash \
+    grep \
+    musl \
+    packagegroup-core-buildessential \
+    sed \
+"
+
+do_install_ptest () {
+    cp ${S}/Makefile ${D}${PTEST_PATH}
+    cp ${S}/config.mak.def ${D}${PTEST_PATH}/config.mak
+    cp -r ${S}/src ${D}${PTEST_PATH}/
+}
+
+# libc-test-ptest depends on empty libc-test.
+ALLOW_EMPTY:${PN} = "1"
+
+COMPATIBLE_HOST = "null"
+COMPATIBLE_HOST:libc-musl = "(.*)"
-- 
2.25.1

Re: [OE-core] [PATCH] libc-test: add libc testsuite for musl

[Alexandre Belloni]

Hello,

On 03/11/2022 05:39:26+0000, Chase Qi wrote:
> libc-test is a collection of unit test to measure the correctness and
> robustness of a C/POSIX standard library implementation. It is developed
> as part of the musl project.
> 

You also need to add it to the ptest list:

WARNING: libc-test-git-r0 do_package_qa: QA Issue: supports ptests but is not included in oe-core's ptest-packagelists.inc [missing-ptest]

> Signed-off-by: Chase Qi <chase.qi@linaro.org>
> ---
>  meta/recipes-core/musl/libc-test/run-ptest | 17 ++++++++
>  meta/recipes-core/musl/libc-test_git.bb    | 45 ++++++++++++++++++++++
>  2 files changed, 62 insertions(+)
>  create mode 100644 meta/recipes-core/musl/libc-test/run-ptest
>  create mode 100644 meta/recipes-core/musl/libc-test_git.bb
> 
> diff --git a/meta/recipes-core/musl/libc-test/run-ptest b/meta/recipes-core/musl/libc-test/run-ptest
> new file mode 100644
> index 0000000000..30abcdfba5
> --- /dev/null
> +++ b/meta/recipes-core/musl/libc-test/run-ptest
> @@ -0,0 +1,17 @@
> +#!/bin/sh
> +
> +make cleanall
> +make run | tee libc-test.log
> +
> +echo ""
> +echo "--- ptest result ---"
> +if grep -q '^FAIL src.*\.exe.*' libc-test.log; then
> +    # Consolidate output format.
> +    # e.g. "FAIL src/api/main.exe [status 1]" -> "FAIL: api_main"
> +    grep '^FAIL src.*\.exe.*' libc-test.log \
> +        | sed 's|^FAIL src/|FAIL: |;s|/|_|;s|\.exe.*\]||'
> +    exit 1
> +else
> +    echo "PASS: libc-test"
> +    exit 0
> +fi
> diff --git a/meta/recipes-core/musl/libc-test_git.bb b/meta/recipes-core/musl/libc-test_git.bb
> new file mode 100644
> index 0000000000..d5d8845727
> --- /dev/null
> +++ b/meta/recipes-core/musl/libc-test_git.bb
> @@ -0,0 +1,45 @@
> +SUMMARY = "Musl libc test to be run with ptest"
> +HOMEPAGE = "https://wiki.musl-libc.org/libc-test.html"
> +DESCRIPTION = "libc-test is a collection of unit tests to measure the \
> +correctness and robustness of a C/POSIX standard library implementation. It is \
> +developed as part of the musl project."
> +SECTION = "tests"
> +LICENSE = "MIT"
> +LIC_FILES_CHKSUM = "file://COPYRIGHT;md5=43ed1245085be90dc934288117d55a3b"
> +
> +inherit ptest
> +
> +SRCREV = "18e28496adee3d84fefdda6efcb9c5b8996a2398"
> +SRC_URI = " \
> +    git://nsz.repo.hu:49100/repo/libc-test;branch=master \
> +    file://run-ptest \
> +"
> +MIRRORS += "git://nsz.repo.hu:49100/repo/libc-test git://repo.or.cz/libc-test"
> +
> +S = "${WORKDIR}/git"
> +
> +# libc-test 'make' or 'make run' command is designed to build and run tests. It
> +# reports both build and test failures. The commands should be run on target.
> +do_compile() {
> +    :
> +}
> +
> +RDEPENDS:${PN}-ptest = " \
> +    bash \
> +    grep \
> +    musl \
> +    packagegroup-core-buildessential \
> +    sed \
> +"
> +
> +do_install_ptest () {
> +    cp ${S}/Makefile ${D}${PTEST_PATH}
> +    cp ${S}/config.mak.def ${D}${PTEST_PATH}/config.mak
> +    cp -r ${S}/src ${D}${PTEST_PATH}/
> +}
> +
> +# libc-test-ptest depends on empty libc-test.
> +ALLOW_EMPTY:${PN} = "1"
> +
> +COMPATIBLE_HOST = "null"
> +COMPATIBLE_HOST:libc-musl = "(.*)"
> -- 
> 2.25.1
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#172610): https://lists.openembedded.org/g/openembedded-core/message/172610
> Mute This Topic: https://lists.openembedded.org/mt/94751275/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH] libc-test: add libc testsuite for musl

[Ross Burton]

On 3 Nov 2022, at 05:39, Chase Qi via lists.openembedded.org <chase.qi=linaro.org@lists.openembedded.org> wrote:
> +MIRRORS += "git://nsz.repo.hu:49100/repo/libc-test git://repo.or.cz/libc-test"

Why do this recipe need a MIRRORS?

> +RDEPENDS:${PN}-ptest = " \
> +    bash \
> +    grep \
> +    musl \
> +    packagegroup-core-buildessential \
> +    sed \
> +"
> +
> +do_install_ptest () {
> +    cp ${S}/Makefile ${D}${PTEST_PATH}
> +    cp ${S}/config.mak.def ${D}${PTEST_PATH}/config.mak
> +    cp -r ${S}/src ${D}${PTEST_PATH}/
> +}
> +
> +# libc-test-ptest depends on empty libc-test.
> +ALLOW_EMPTY:${PN} = "1"

It would be neater if the test suite was installed to libc-test, and libc-test-ptest simply contained the run-ptest script. This way the packaging reflects reality and you don’t need to set ALLOW_EMPTY.

Basically: the package contains a test suite, and a ptest harness.  If we ever move from ptest to something else then the libc-test package won’t change, just the harness will.

> +COMPATIBLE_HOST = "null"
> +COMPATIBLE_HOST:libc-musl = "(.*)"

Is there any value in running the test suite on glibc?

Ross

Re: [OE-core] [PATCH] libc-test: add libc testsuite for musl

[Chase Qi]

Hi Ross,

Thanks for the review.

On Thu, 3 Nov 2022 at 21:51, Ross Burton <Ross.Burton@arm.com> wrote:
>
> On 3 Nov 2022, at 05:39, Chase Qi via lists.openembedded.org <chase.qi=linaro.org@lists.openembedded.org> wrote:
> > +MIRRORS += "git://nsz.repo.hu:49100/repo/libc-test git://repo.or.cz/libc-test"
>
> Why do this recipe need a MIRRORS?
>

The main repo looks like a personal repo hosted in a kind of
self-maintained hosting service. It is owned by http://igor2.repo.hu/.
The mirror is the official mirror documented on the wiki
https://wiki.musl-libc.org/libc-test.html. I think it can be helpful
when the main repo is unreachable.

> > +RDEPENDS:${PN}-ptest = " \
> > +    bash \
> > +    grep \
> > +    musl \
> > +    packagegroup-core-buildessential \
> > +    sed \
> > +"
> > +
> > +do_install_ptest () {
> > +    cp ${S}/Makefile ${D}${PTEST_PATH}
> > +    cp ${S}/config.mak.def ${D}${PTEST_PATH}/config.mak
> > +    cp -r ${S}/src ${D}${PTEST_PATH}/
> > +}
> > +
> > +# libc-test-ptest depends on empty libc-test.
> > +ALLOW_EMPTY:${PN} = "1"
>
> It would be neater if the test suite was installed to libc-test, and libc-test-ptest simply contained the run-ptest script. This way the packaging reflects reality and you don’t need to set ALLOW_EMPTY.
>
> Basically: the package contains a test suite, and a ptest harness.  If we ever move from ptest to something else then the libc-test package won’t change, just the harness will.
>

+1 Thanks for your proposal. I have submitted v2. Would you please take a look?

> > +COMPATIBLE_HOST = "null"
> > +COMPATIBLE_HOST:libc-musl = "(.*)"
>
> Is there any value in running the test suite on glibc?
>

I tried the test suite on glibc. It reported all tests passed, but I
do see errors in the test log. I think it doesn't work well on glibc.
Considering glibc has its own test suite, the
'meta/recipes-core/glibc/glibc-tests_2.36.bb', I think we should start
with musl only.

Thanks,
Chase

> Ross
>