[LTP] [PATCH v2 0/7] CI: Run metadata tests

[LTP] [PATCH v2 0/7] CI: Run metadata tests

[Petr Vorel]

Changes v1->v2:
* Fix out of tree build
* Try to detect out-of-tree build without specifying variable
* Makefile: Skip test-metadata on cross compile (new commit)
* Add missing diffutils for openSUSE
* Add missing licenses, copyright (new commit)

Tested:
https://github.com/pevik/ltp/actions/runs/29044863389

Petr Vorel (8):
  Makefile: Skip test-metadata on cross compile
  metadata: test.sh: Print the test name and result
  metadata: Fix dependency for tests
  metadata: test.sh: Add support for building out-of-tree
  metadata: Add missing licenses, copyright
  build.sh: Add support for test-metadata
  ci/tumbleweed: Install diffutils
  ci: Test metadata in docker build testing

 .github/workflows/ci-docker-build.yml |  5 +++++
 Makefile                              |  5 ++++-
 build.sh                              |  5 +++--
 ci/tumbleweed.sh                      |  3 ++-
 metadata/Makefile                     |  4 ++--
 metadata/tests/Makefile               |  3 +++
 metadata/tests/test.sh                | 19 ++++++++++++++++---
 7 files changed, 35 insertions(+), 9 deletions(-)

-- 
2.54.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 1/8] Makefile: Skip test-metadata on cross compile

[Petr Vorel]

Fixes: 73245ef992 ("make: Add test-metadata target")
Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
* New in v2

 Makefile | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/Makefile b/Makefile
index 2a7cf54caa..f1b59609a5 100644
--- a/Makefile
+++ b/Makefile
@@ -210,6 +210,9 @@ endif
 	$(top_srcdir)/testcases/lib/run_tests.sh -b $(abs_builddir)
 
 test-metadata: metadata-all
+ifneq ($(build),$(host))
+	$(error running tests on cross-compile build not supported)
+endif
 	$(MAKE) -C $(abs_srcdir)/metadata test
 
 MODULE_DIRS :=  $(shell \
-- 
2.54.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 2/8] metadata: test.sh: Print the test name and result

[Petr Vorel]

Always print the test name with the result. This helps a readability
in the CI.

    * multiline_macro.c [OK]
    * regression.c [FAIL]
    regression.c output differs!
    --- tmp.json	2026-07-09 14:27:15.590074023 +0200
    +++ regression.c.json	2026-07-09 14:27:06.083967188 +0200
    @@ -6,7 +6,7 @@
	 ]
	],
	"doc": [
    -    "Test for regression group derived from the linux-git tag."
    +    "Test for regression group derived from the foo tag."
	],
	"groups": [
	 "regression"
    * tags.c [OK]

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
* The same as in v1

 metadata/tests/test.sh | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/metadata/tests/test.sh b/metadata/tests/test.sh
index 475d721df0..a9ebc768f9 100755
--- a/metadata/tests/test.sh
+++ b/metadata/tests/test.sh
@@ -3,13 +3,15 @@
 fail=0
 
 for i in *.c; do
+	printf "* $i "
 	../metaparse $i > tmp.json
 	if ! diff tmp.json $i.json >/dev/null 2>&1; then
-		echo "***"
+		echo '[FAIL]'
 		echo "$i output differs!"
 		diff -u tmp.json $i.json
-		echo "***"
 		fail=1
+	else
+		echo '[OK]'
 	fi
 done
 
-- 
2.54.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 3/8] metadata: Fix dependency for tests

[Petr Vorel]

'tests' target needs to have metadata binaries being built.

Fixes: af905936db ("docparse: Split into metadata and docparse")
Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
* New in v2

 metadata/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/metadata/Makefile b/metadata/Makefile
index 6939b9f76c..7e4ee38d2e 100644
--- a/metadata/Makefile
+++ b/metadata/Makefile
@@ -15,7 +15,7 @@ INSTALL_DIR		= metadata
 ltp.json: metaparse metaparse-sh
 	$(abs_srcdir)/parse.sh > ltp.json
 
-test:
+test: metaparse metaparse-sh
 	$(MAKE) -C $(abs_srcdir)/tests/ test
 
 include $(top_srcdir)/include/mk/generic_leaf_target.mk
-- 
2.54.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 4/8] metadata: test.sh: Add support for building out-of-tree

[Petr Vorel]

NOTE out-of-tree build is supported only from the top level directory,
otherwise it requires to set METAPARSEDIR variable by user (similarly to
other tests run via testcases/lib/run_tests.sh).

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
Changes v1->v2:
* Fix out of tree build
* Try to detect out-of-tree build without specifying variable

LTP running internal testing is implemented for every directory
differently. It would be nice to unify it.

Also much of the complexity is done by out-of-tree part. I really wish
one day we migrate to meson (which is faster, support out-of-tree
modules and de facto standard for userspace nowadays).

 Makefile               |  2 +-
 metadata/Makefile      |  2 +-
 metadata/tests/test.sh | 11 ++++++++++-
 3 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index f1b59609a5..2efbe8c6c0 100644
--- a/Makefile
+++ b/Makefile
@@ -213,7 +213,7 @@ test-metadata: metadata-all
 ifneq ($(build),$(host))
 	$(error running tests on cross-compile build not supported)
 endif
-	$(MAKE) -C $(abs_srcdir)/metadata test
+	METAPARSEDIR=$(abs_builddir)/metadata $(MAKE) -C $(abs_srcdir)/metadata test
 
 MODULE_DIRS :=  $(shell \
 	dirname $$(grep -l 'include.*module\.mk' $$(find $(abs_srcdir)/testcases/ -type f -name 'Makefile')))
diff --git a/metadata/Makefile b/metadata/Makefile
index 7e4ee38d2e..af194bcc94 100644
--- a/metadata/Makefile
+++ b/metadata/Makefile
@@ -16,6 +16,6 @@ ltp.json: metaparse metaparse-sh
 	$(abs_srcdir)/parse.sh > ltp.json
 
 test: metaparse metaparse-sh
-	$(MAKE) -C $(abs_srcdir)/tests/ test
+	METAPARSEDIR=$$METAPARSEDIR $(MAKE) -C tests/ test
 
 include $(top_srcdir)/include/mk/generic_leaf_target.mk
diff --git a/metadata/tests/test.sh b/metadata/tests/test.sh
index a9ebc768f9..89a7857ad2 100755
--- a/metadata/tests/test.sh
+++ b/metadata/tests/test.sh
@@ -2,9 +2,18 @@
 
 fail=0
 
+METAPARSE="${METAPARSEDIR:-..}/metaparse"
+
+cd "${SRCDIR:-.}"
+
+if [ ! -x "$METAPARSE" ]; then
+	echo "Error: $METAPARSE not found (wrong PATH? out-of-tree build without specifying \$METAPARSEDIR?)" >&2
+	exit 1
+fi
+
 for i in *.c; do
 	printf "* $i "
-	../metaparse $i > tmp.json
+	$METAPARSE $i > tmp.json
 	if ! diff tmp.json $i.json >/dev/null 2>&1; then
 		echo '[FAIL]'
 		echo "$i output differs!"
-- 
2.54.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 5/8] metadata: Add missing licenses, copyright

[Petr Vorel]

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
* New in v2

 metadata/tests/Makefile | 3 +++
 metadata/tests/test.sh  | 2 ++
 2 files changed, 5 insertions(+)

diff --git a/metadata/tests/Makefile b/metadata/tests/Makefile
index b5c8c46686..ade3b85977 100644
--- a/metadata/tests/Makefile
+++ b/metadata/tests/Makefile
@@ -1,3 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (c) Linux Test Project, 2021-2026
+
 all:
 
 test:
diff --git a/metadata/tests/test.sh b/metadata/tests/test.sh
index 89a7857ad2..e52461f6f4 100755
--- a/metadata/tests/test.sh
+++ b/metadata/tests/test.sh
@@ -1,4 +1,6 @@
 #!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (c) Linux Test Project, 2021-2026
 
 fail=0
 
-- 
2.54.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 6/8] build.sh: Add support for test-metadata

[Petr Vorel]

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
* The same as in v1

 build.sh | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/build.sh b/build.sh
index 47a5a7b050..57bb9e904c 100755
--- a/build.sh
+++ b/build.sh
@@ -178,6 +178,7 @@ configure          run only 'configure'
 build              run only 'make'
 test               run only 'make test' (not supported for cross-compile build)
 test-c             run only 'make test-c' (not supported for cross-compile build)
+test-metadata      run only 'make test-metadata' (not supported for cross-compile build)
 test-shell         run only 'make test-shell' (not supported for cross-compile build)
 test-shell-loader  run only 'make test-shell-loader' (not supported for cross-compile build)
 install            run only 'make install'
@@ -207,7 +208,7 @@ while getopts "c:hio:p:r:t:" opt; do
 		esac;;
 	p) prefix="$OPTARG";;
 	r) case "$OPTARG" in
-		autotools|configure|build|test|test-c|test-shell|test-shell-loader|install) run="$OPTARG";;
+		autotools|configure|build|test|test-c|test-metadata|test-shell|test-shell-loader|install) run="$OPTARG";;
 		*) echo "Wrong run type '$OPTARG'" >&2; usage; exit 1;;
 		esac;;
 	t) case "$OPTARG" in
@@ -233,7 +234,7 @@ if [ -z "$run" -o "$run" = "build" ]; then
 	eval build_${tree}_tree
 fi
 
-if [ -z "$run" -o "$run" = "test" -o "$run" = "test-c" -o "$run" = "test-shell" -o "$run" = "test-shell-loader" ]; then
+if [ -z "$run" -o "$run" = "test" -o "$run" = "test-c" -o "$run" = "test-metadata" -o "$run" = "test-shell" -o "$run" = "test-shell-loader" ]; then
 	if [ "$build" = "cross" ]; then
 		echo "cross-compile build, skipping running tests" >&2
 	else
-- 
2.54.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 7/8] ci/tumbleweed: Install diffutils

[Petr Vorel]

This is needed at least for openSUSE Leap 16.
It will be needed for running metadata tests.

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
* New in v2

 ci/tumbleweed.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/ci/tumbleweed.sh b/ci/tumbleweed.sh
index 8f23229dfc..94c4d719d1 100755
--- a/ci/tumbleweed.sh
+++ b/ci/tumbleweed.sh
@@ -1,6 +1,6 @@
 #!/bin/sh -eux
 # SPDX-License-Identifier: GPL-2.0-or-later
-# Copyright (c) 2018-2025 Petr Vorel <pvorel@suse.cz>
+# Copyright (c) 2018-2026 Petr Vorel <pvorel@suse.cz>
 
 zyp="zypper --non-interactive install --force-resolution --no-recommends"
 
@@ -11,6 +11,7 @@ while [ $i != 0 ]; do
 		automake \
 		clang \
 		curl \
+		diffutils \
 		jq \
 		findutils \
 		gcc \
-- 
2.54.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v2 8/8] ci: Test metadata in docker build testing

[Petr Vorel]

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
* The same as in v1

 .github/workflows/ci-docker-build.yml | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/.github/workflows/ci-docker-build.yml b/.github/workflows/ci-docker-build.yml
index 405a7f0690..52b8dc4090 100644
--- a/.github/workflows/ci-docker-build.yml
+++ b/.github/workflows/ci-docker-build.yml
@@ -178,6 +178,11 @@ jobs:
         case "$VARIANT" in cross-compile*) BUILD="cross";; i386) BUILD="32";; *) BUILD="native";; esac
         ./build.sh -r test-c -o ${TREE:-in} -t $BUILD
 
+    - name: Test metadata
+      run: |
+        case "$VARIANT" in cross-compile*) BUILD="cross";; i386) BUILD="32";; *) BUILD="native";; esac
+        ./build.sh -r test-metadata -o ${TREE:-in} -t $BUILD
+
     - name: Test shell API
       run: |
         case "$VARIANT" in cross-compile*) BUILD="cross";; i386) BUILD="32";; *) BUILD="native";; esac
-- 
2.54.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 2/8] metadata: test.sh: Print the test name and result

[Petr Vorel]

Hi,

> +++ b/metadata/tests/test.sh
> @@ -3,13 +3,15 @@
>  fail=0

>  for i in *.c; do
> +	printf "* $i "

OK, I will follow the CI advice from v1 and change before merge:
-       printf "* $i "
+       printf '* %s ' "$i"

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] Makefile: Skip test-metadata on cross compile

[linuxtestproject.agent]

Hi Petr,

On Fri, 10 Jul 2026 07:55:49 +0200, Petr Vorel wrote:
> Makefile: Skip test-metadata on cross compile

Verdict - Reviewed

---
Note:

The agent can sometimes produce false positives although often its
findings are genuine. If you find issues with the review, please
comment this email or ignore the suggestions.

Regards,
LTP AI Reviewer

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] Makefile: Skip test-metadata on cross compile

[linuxtestproject.agent]

Hi Petr,

On Fri, 10 Jul 2026 07:55:49 +0200, Petr Vorel wrote:
> Makefile: Skip test-metadata on cross compile

--- [PATCH 2/8] ---

> +	printf "* $i "

$i is used directly as part of the printf format string rather than
passed as a separate argument.  A filename containing a '%' character
would be interpreted as a format specifier.

The portable form is:

    printf "* %s " "$i"

--- [PATCH 4/8] ---

> +	$METAPARSE $i > tmp.json

Both $METAPARSE and $i are unquoted.  If METAPARSEDIR (and therefore
METAPARSE) contains spaces, the shell will perform word-splitting on
the variable before exec, breaking the invocation.

The quoted form:

    "$METAPARSE" "$i" > tmp.json

Verdict - Needs revision

---
Note:

The agent can sometimes produce false positives although often its
findings are genuine. If you find issues with the review, please
comment this email or ignore the suggestions.

Regards,
LTP AI Reviewer

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 4/8] metadata: test.sh: Add support for building out-of-tree

[Cyril Hrubis]

Hi!
We keep fixing the out-of-tree build but is it worth the amout of work
we put into it? Is that even used? Wouldn't it make more sense to strip
it completely, which would simplify our build system greatly?

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 4/8] metadata: test.sh: Add support for building out-of-tree

[Petr Vorel]

Hi Cyril, Martin, Andrea,

> Hi!
> We keep fixing the out-of-tree build but is it worth the amout of work
> we put into it? Is that even used? Wouldn't it make more sense to strip
> it completely, which would simplify our build system greatly?

I remember Martin asked to keep it because he's using it.

I would say no, it's not worth of that trouble. I would either live without
out-of-tree build or reevaluate what is blocking for switching to meson.
(I don't remember what Andrea found in his POC some years ago.)

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 4/8] metadata: test.sh: Add support for building out-of-tree

[Cyril Hrubis]

Hi!
> I would say no, it's not worth of that trouble. I would either live without
> out-of-tree build or reevaluate what is blocking for switching to meson.
> (I don't remember what Andrea found in his POC some years ago.)

I'm not 100% convinced that meson is the best way to go. It increases
the build dependecies quite a lot from a autotools+make to python with
several libraries.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 4/8] metadata: test.sh: Add support for building out-of-tree

[Andrea Cervesato via ltp]

Hi Cyril,

> Hi!
> > I would say no, it's not worth of that trouble. I would either live without
> > out-of-tree build or reevaluate what is blocking for switching to meson.
> > (I don't remember what Andrea found in his POC some years ago.)
> 
> I'm not 100% convinced that meson is the best way to go. It increases
> the build dependecies quite a lot from a autotools+make to python with
> several libraries.

I understand your point of view, but meson doesn't have dependencies
beside Python 3.10+ (pure python) and Ninja.

So we would switch from make + autotools to python + ninja. Still,
2 tools. The only difference is that make + autotools is messy and
legacy, while Meson is modern and well crafted.

The project I worked on is easily adaptable in the current LTP version:
https://github.com/acerv/ltp-core

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 4/8] metadata: test.sh: Add support for building out-of-tree

[Cyril Hrubis]

Hi!
> I understand your point of view, but meson doesn't have dependencies
> beside Python 3.10+ (pure python) and Ninja.
> 
> So we would switch from make + autotools to python + ninja. Still,
> 2 tools. The only difference is that make + autotools is messy and
> legacy, while Meson is modern and well crafted.
> 
> The project I worked on is easily adaptable in the current LTP version:
> https://github.com/acerv/ltp-core

Dependencies aside another question is how will it scale for the whole
LTP. We have thousands of test binaries to build, these would need to be
written as targets somewhere, hopefully in separate files per test
directory and then somehow included in the top level meson
configuration. With make the targets are mostly autogenerated. And what
about 'make check' and other tooling we have?

Writing a prototype for the whole LTP would be a lot of work and it may
not even produce something that is usable.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 4/8] metadata: test.sh: Add support for building out-of-tree

[Andrea Cervesato via ltp]

> Writing a prototype for the whole LTP would be a lot of work and it may
> not even produce something that is usable.

This is questionable, I think it would take no more than half a day with
the LLMs applied to the PoC I wrote a couple of years ago. I can try it
in the free time and see how it looks like.

When it's available I will send you a draft so you can take a look at it
and we can evaluate if it makes sense to try or not. I'm sure there are
corner cases where autotools is better and vice versa.

Regards,
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 2/8] metadata: test.sh: Print the test name and result

[Andrea Cervesato via ltp]

Reviewed-by: Andrea Cervesato <andrea.cervesato@suse.com>

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 3/8] metadata: Fix dependency for tests

[Andrea Cervesato via ltp]

Reviewed-by: Andrea Cervesato <andrea.cervesato@suse.com>

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 4/8] metadata: test.sh: Add support for building out-of-tree

[Andrea Cervesato via ltp]

Hi Petr,

here I agree with Cyril, I don't think we need to fix out-of-tree
builds. We need to run make inside the folder from the very
beginning and, since this is a requirement, it makes sense to
avoid fixing these use cases.

Regards,
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 5/8] metadata: Add missing licenses, copyright

[Andrea Cervesato via ltp]

Reviewed-by: Andrea Cervesato <andrea.cervesato@suse.com>

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 6/8] build.sh: Add support for test-metadata

[Andrea Cervesato via ltp]

Hi Petr,

> -if [ -z "$run" -o "$run" = "test" -o "$run" = "test-c" -o "$run" = "test-shell" -o "$run" = "test-shell-loader" ]; then
> +if [ -z "$run" -o "$run" = "test" -o "$run" = "test-c" -o "$run" = "test-metadata" -o "$run" = "test-shell" -o "$run" = "test-shell-loader" ]; then

This is getting uglier. What about this?

if [ -z "$run" ] || [ "${run#test}" != "$run" ]; then

The rest LGTM:

Reviewed-by: Andrea Cervesato <andrea.cervesato@suse.com>

Regards,
--
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 7/8] ci/tumbleweed: Install diffutils

[Andrea Cervesato via ltp]

Reviewed-by: Andrea Cervesato <andrea.cervesato@suse.com>

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 8/8] ci: Test metadata in docker build testing

[Andrea Cervesato via ltp]

Reviewed-by: Andrea Cervesato <andrea.cervesato@suse.com>

--
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 4/8] metadata: test.sh: Add support for building out-of-tree

[Cyril Hrubis]

Hi!
> This is questionable, I think it would take no more than half a day with
> the LLMs applied to the PoC I wrote a couple of years ago. I can try it
> in the free time and see how it looks like.

That's certainly true.

> When it's available I will send you a draft so you can take a look at it
> and we can evaluate if it makes sense to try or not. I'm sure there are
> corner cases where autotools is better and vice versa.

Feel free. From what I've seen the modern make replacements were not
working that well with projects that produces many small binaries. But
that was a few years ago, maybe they are better now.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp