[LTP] [PATCH v2] lib: tst_test: Add reproducible output.

[LTP] [PATCH v2] lib: tst_test: Add reproducible output.

[Cyril Hrubis]

This commit adds an environment variable LTP_REPRODUCIBLE_OUTPUT that
when set skips printing parts of the test messages that may contain data
that differ on subsequent runs (e.g. pids).

With this you can run a test twice under a different conditions and
check if the test codeflow was identical by simply doing diff of the
outputs from the two runs.

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
Suggested-by: Martin Doucha <mdoucha@suse.cz>
CC: valgrind-developers@lists.sourceforge.net
---
 lib/tst_test.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/lib/tst_test.c b/lib/tst_test.c
index 2bb4519dd..f14627544 100644
--- a/lib/tst_test.c
+++ b/lib/tst_test.c
@@ -64,6 +64,7 @@ static int mntpoint_mounted;
 static int ovl_mounted;
 static struct timespec tst_start_time; /* valid only for test pid */
 static int tdebug;
+static int reproducible_output;
 
 struct results {
 	int passed;
@@ -312,6 +313,9 @@ static void print_result(const char *file, const int lineno, int ttype,
 	str += ret;
 	size -= ret;
 
+	if (reproducible_output)
+		goto print;
+
 	ssize = size - 2;
 	ret = vsnprintf(str, size, fmt, va);
 	str += MIN(ret, ssize);
@@ -329,6 +333,7 @@ static void print_result(const char *file, const int lineno, int ttype,
 				"Next message is too long and truncated:");
 	}
 
+print:
 	snprintf(str, size, "\n");
 
 	/* we might be called from signal handler, so use write() */
@@ -1312,6 +1317,8 @@ static void do_setup(int argc, char *argv[])
 	if (tst_test->supported_archs && !tst_is_on_arch(tst_test->supported_archs))
 		tst_brk(TCONF, "This arch '%s' is not supported for test!", tst_arch.name);
 
+	reproducible_output = !!getenv("LTP_REPRODUCIBLE_OUTPUT");
+
 	assert_test_fn();
 
 	TCID = tid = get_tid(argv);
-- 
2.49.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] lib: tst_test: Add reproducible output.

[Martin Doucha]

Hi,
Reviewed-by: Martin Doucha <mdoucha@suse.cz>

On 09. 05. 25 11:28, Cyril Hrubis wrote:
> This commit adds an environment variable LTP_REPRODUCIBLE_OUTPUT that
> when set skips printing parts of the test messages that may contain data
> that differ on subsequent runs (e.g. pids).
> 
> With this you can run a test twice under a different conditions and
> check if the test codeflow was identical by simply doing diff of the
> outputs from the two runs.
> 
> Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> Suggested-by: Martin Doucha <mdoucha@suse.cz>
> CC: valgrind-developers@lists.sourceforge.net
> ---
>   lib/tst_test.c | 7 +++++++
>   1 file changed, 7 insertions(+)
> 
> diff --git a/lib/tst_test.c b/lib/tst_test.c
> index 2bb4519dd..f14627544 100644
> --- a/lib/tst_test.c
> +++ b/lib/tst_test.c
> @@ -64,6 +64,7 @@ static int mntpoint_mounted;
>   static int ovl_mounted;
>   static struct timespec tst_start_time; /* valid only for test pid */
>   static int tdebug;
> +static int reproducible_output;
>   
>   struct results {
>   	int passed;
> @@ -312,6 +313,9 @@ static void print_result(const char *file, const int lineno, int ttype,
>   	str += ret;
>   	size -= ret;
>   
> +	if (reproducible_output)
> +		goto print;
> +
>   	ssize = size - 2;
>   	ret = vsnprintf(str, size, fmt, va);
>   	str += MIN(ret, ssize);
> @@ -329,6 +333,7 @@ static void print_result(const char *file, const int lineno, int ttype,
>   				"Next message is too long and truncated:");
>   	}
>   
> +print:
>   	snprintf(str, size, "\n");
>   
>   	/* we might be called from signal handler, so use write() */
> @@ -1312,6 +1317,8 @@ static void do_setup(int argc, char *argv[])
>   	if (tst_test->supported_archs && !tst_is_on_arch(tst_test->supported_archs))
>   		tst_brk(TCONF, "This arch '%s' is not supported for test!", tst_arch.name);
>   
> +	reproducible_output = !!getenv("LTP_REPRODUCIBLE_OUTPUT");
> +
>   	assert_test_fn();
>   
>   	TCID = tid = get_tid(argv);


-- 
Martin Doucha   mdoucha@suse.cz
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] lib: tst_test: Add reproducible output.

[Avinesh Kumar]

Hi,

Reviewed-by: Avinesh Kumar <akumar@suse.de>


On Friday, May 9, 2025 11:28:13 AM CEST Cyril Hrubis wrote:
> This commit adds an environment variable LTP_REPRODUCIBLE_OUTPUT that
> when set skips printing parts of the test messages that may contain data
> that differ on subsequent runs (e.g. pids).
> 
> With this you can run a test twice under a different conditions and
> check if the test codeflow was identical by simply doing diff of the
> outputs from the two runs.
> 
> Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> Suggested-by: Martin Doucha <mdoucha@suse.cz>
> CC: valgrind-developers@lists.sourceforge.net
> ---
>  lib/tst_test.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/lib/tst_test.c b/lib/tst_test.c
> index 2bb4519dd..f14627544 100644
> --- a/lib/tst_test.c
> +++ b/lib/tst_test.c
> @@ -64,6 +64,7 @@ static int mntpoint_mounted;
>  static int ovl_mounted;
>  static struct timespec tst_start_time; /* valid only for test pid */
>  static int tdebug;
> +static int reproducible_output;
>  
>  struct results {
>  	int passed;
> @@ -312,6 +313,9 @@ static void print_result(const char *file, const int lineno, int ttype,
>  	str += ret;
>  	size -= ret;
>  
> +	if (reproducible_output)
> +		goto print;
> +
>  	ssize = size - 2;
>  	ret = vsnprintf(str, size, fmt, va);
>  	str += MIN(ret, ssize);
> @@ -329,6 +333,7 @@ static void print_result(const char *file, const int lineno, int ttype,
>  				"Next message is too long and truncated:");
>  	}
>  
> +print:
>  	snprintf(str, size, "\n");
>  
>  	/* we might be called from signal handler, so use write() */
> @@ -1312,6 +1317,8 @@ static void do_setup(int argc, char *argv[])
>  	if (tst_test->supported_archs && !tst_is_on_arch(tst_test->supported_archs))
>  		tst_brk(TCONF, "This arch '%s' is not supported for test!", tst_arch.name);
>  
> +	reproducible_output = !!getenv("LTP_REPRODUCIBLE_OUTPUT");
> +
>  	assert_test_fn();
>  
>  	TCID = tid = get_tid(argv);
> 

Regards,
Avinesh



-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] lib: tst_test: Add reproducible output.

[Petr Vorel]

Hi Cyril,

> This commit adds an environment variable LTP_REPRODUCIBLE_OUTPUT that
> when set skips printing parts of the test messages that may contain data
> that differ on subsequent runs (e.g. pids).

> With this you can run a test twice under a different conditions and
> check if the test codeflow was identical by simply doing diff of the
> outputs from the two runs.

LGTM, thanks!
Reviewed-by: Petr Vorel <pvorel@suse.cz>

Could you please add before the commit the variable to
print_help() and to the docs doc/users/setup_tests.rst?

I suggest the diff below, which also sort the entries (ideally it'd be
separated, but it's before the release, feel free to do it).

> Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> Suggested-by: Martin Doucha <mdoucha@suse.cz>
> CC: valgrind-developers@lists.sourceforge.net
> ---
>  lib/tst_test.c | 7 +++++++
>  1 file changed, 7 insertions(+)

> diff --git a/lib/tst_test.c b/lib/tst_test.c
> index 2bb4519dd..f14627544 100644
> --- a/lib/tst_test.c
> +++ b/lib/tst_test.c
> @@ -64,6 +64,7 @@ static int mntpoint_mounted;
>  static int ovl_mounted;
>  static struct timespec tst_start_time; /* valid only for test pid */
>  static int tdebug;
> +static int reproducible_output;

>  struct results {
>  	int passed;
> @@ -312,6 +313,9 @@ static void print_result(const char *file, const int lineno, int ttype,
>  	str += ret;
>  	size -= ret;

> +	if (reproducible_output)
> +		goto print;
> +
>  	ssize = size - 2;
>  	ret = vsnprintf(str, size, fmt, va);
>  	str += MIN(ret, ssize);
> @@ -329,6 +333,7 @@ static void print_result(const char *file, const int lineno, int ttype,
>  				"Next message is too long and truncated:");
>  	}

> +print:
>  	snprintf(str, size, "\n");

>  	/* we might be called from signal handler, so use write() */
> @@ -1312,6 +1317,8 @@ static void do_setup(int argc, char *argv[])
>  	if (tst_test->supported_archs && !tst_is_on_arch(tst_test->supported_archs))
>  		tst_brk(TCONF, "This arch '%s' is not supported for test!", tst_arch.name);

> +	reproducible_output = !!getenv("LTP_REPRODUCIBLE_OUTPUT");

LTP_COLORIZE_OUTPUT allows more complicated setup (y/1 always, n/0: never),
but any value here enables the feature (e.g. LTP_COLORIZE_OUTPUT=0).
This would be nice to unify, but I'm ok to ignore it as it's before release.

> +
>  	assert_test_fn();

>  	TCID = tid = get_tid(argv);

Adding docs (+ sort variables while at it).

Kind regards,
Petr

+++ doc/users/setup_tests.rst
@@ -49,11 +49,12 @@ users.
    * - LTP_DEV_FS_TYPE
      - Filesystem used for testing (default: ``ext2``).
 
-   * - LTP_TIMEOUT_MUL
-     - Multiplies timeout, must be number >= 0.1 (> 1 is useful for slow
-       machines to avoid unexpected timeout). It's mainly for shell API, which
-       does not have LTP_RUNTIME_MUL. In C API it scales the default 30 sec
-       safety margin, probably LTP_RUNTIME_MUL should be used instead.
+   * - LTP_IMA_LOAD_POLICY
+     - Load IMA example policy, see :master:`testcases/kernel/security/integrity/ima/README.md`.
+
+   * - LTP_REPRODUCIBLE_OUTPUT
+     - Dischard the actual content of the messages printed by the test
+       (suitable for a reproducible output).
 
    * - LTP_RUNTIME_MUL
      - Multiplies maximal test iteration runtime. Tests that run for more than a
@@ -61,8 +62,11 @@ users.
        both up and down with this multiplier. This is not yet implemented in the
        shell API.
 
-   * - LTP_IMA_LOAD_POLICY
-     - Load IMA example policy, see :master:`testcases/kernel/security/integrity/ima/README.md`.
+   * - LTP_TIMEOUT_MUL
+     - Multiplies timeout, must be number >= 0.1 (> 1 is useful for slow
+       machines to avoid unexpected timeout). It's mainly for shell API, which
+       does not have LTP_RUNTIME_MUL. In C API it scales the default 30 sec
+       safety margin, probably LTP_RUNTIME_MUL should be used instead.
 
    * - LTP_VIRT_OVERRIDE
      - Overrides virtual machine detection in the test library. Setting it to
diff --git lib/tst_test.c lib/tst_test.c
index f14627544f..a014ddc278 100644
--- lib/tst_test.c
+++ lib/tst_test.c
@@ -613,9 +613,10 @@ static void print_help(void)
 	fprintf(stderr, "LTP_COLORIZE_OUTPUT  Force colorized output behaviour (y/1 always, n/0: never)\n");
 	fprintf(stderr, "LTP_DEV              Path to the block device to be used (for .needs_device)\n");
 	fprintf(stderr, "LTP_DEV_FS_TYPE      Filesystem used for testing (default: %s)\n", DEFAULT_FS_TYPE);
+	fprintf(stderr, "LTP_REPRODUCIBLE_OUTPUT      Dischard the actual content of the messages printed by the test\n");
+	fprintf(stderr, "LTP_RUNTIME_MUL      Runtime multiplier (must be a number >=1)\n");
 	fprintf(stderr, "LTP_SINGLE_FS_TYPE   Testing only - specifies filesystem instead all supported (for .all_filesystems)\n");
 	fprintf(stderr, "LTP_TIMEOUT_MUL      Timeout multiplier (must be a number >=1)\n");
-	fprintf(stderr, "LTP_RUNTIME_MUL      Runtime multiplier (must be a number >=1)\n");
 	fprintf(stderr, "LTP_VIRT_OVERRIDE    Overrides virtual machine detection (values: \"\"|kvm|microsoft|xen|zvm)\n");
 	fprintf(stderr, "TMPDIR               Base directory for template directory (for .needs_tmpdir, default: %s)\n", TEMPDIR);
 	fprintf(stderr, "\n");

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] lib: tst_test: Add reproducible output.

[Cyril Hrubis]

Hi!
> LTP_COLORIZE_OUTPUT allows more complicated setup (y/1 always, n/0: never),
> but any value here enables the feature (e.g. LTP_COLORIZE_OUTPUT=0).
> This would be nice to unify, but I'm ok to ignore it as it's before release.

We also have LTP_ENABLE_DEBUG that has the same values so let's keep the
API the same. I've send v3 with this and the docs changes.

> > +
> >  	assert_test_fn();
> 
> >  	TCID = tid = get_tid(argv);
> 
> Adding docs (+ sort variables while at it).

I've only added the docs for LTP_REPRODUCIBLE_OUTPUT in v3, the sorting
should go in a separate patch. We also miss at least TST_ENABLE_DEBUG in
the output of -h switch as well.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] lib: tst_test: Add reproducible output.

[Martin Cermak via ltp]

On  Thu  2025-05-15  12:34 , Cyril Hrubis wrote:
> Hi!
> > LTP_COLORIZE_OUTPUT allows more complicated setup (y/1 always, n/0: never),
> > but any value here enables the feature (e.g. LTP_COLORIZE_OUTPUT=0).
> > This would be nice to unify, but I'm ok to ignore it as it's before release.
> 
> We also have LTP_ENABLE_DEBUG that has the same values so let's keep the
> API the same. I've send v3 with this and the docs changes.
> 
> > > +
> > >  	assert_test_fn();
> > 
> > >  	TCID = tid = get_tid(argv);
> > 
> > Adding docs (+ sort variables while at it).
> 
> I've only added the docs for LTP_REPRODUCIBLE_OUTPUT in v3, the sorting
> should go in a separate patch. We also miss at least TST_ENABLE_DEBUG in
> the output of -h switch as well.

Thank-you, guys.  I've added a preparation to the valgrind-ltp tester:

https://sourceware.org/git/?p=valgrind.git;a=blob;f=auxprogs/ltp-tester.sh;h=036f196ce21ae6cc692de205ab49774638cfb925;hb=5894abc5fa9de30c6b4dde453bff3ac1034aa330#l19

Once the new version of LTP ships, we can try dropping our workarounds.

Cheers,
Martin


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp