[LTP] [RFC] Reduce .runtime for Long-Running Tests ?

[LTP] [RFC] Reduce .runtime for Long-Running Tests ?

[Li Wang via ltp]

Hello All,

After reviewing some LTP HTML test reports, I noticed that the ten tests
alone take nearly 20 minutes to complete. For example:

--------------------
bind06           300.15s
msgstress01      180.44s
fsx22            170.47s
pty07            150.04s
pty06            150.02s
gf18             121.09s
gf17             120.82s
gf16             120.13s
dirtyc0w_shmem   120.11s
setsockopt07      76.47s

In total, running the full ltp-lite suite currently takes ~1h20m, which is a bit
long for CI or pre-merge validation pipelines.

I'm wondering whether all these .runtime values are truly necessary to reproduce
the intended issues (e.g., race conditions, fuzzing, sync timing issues).
Or if we could:
 - Set a lower threshold for .runtime on general-purpose stress/fuzz tests
 - Introduce a runtime _policy_ for different environments (e.g., fast
CI vs. full weekly runs)

It might be beneficial to revisit the .runtime values of long-running tests and
set a minimal yet effective duration that balances reproducibility with resource
efficiency. This could help save time and free up test resources earlier.

Any thoughts/comments would be appreciated.

-- 
Regards,
Li Wang


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC] Reduce .runtime for Long-Running Tests ?

[Petr Vorel]

Hi Li,

> Hello All,

> After reviewing some LTP HTML test reports, I noticed that the ten tests
> alone take nearly 20 minutes to complete. For example:

> --------------------
> bind06           300.15s

This uses fuzzy sync, I wonder if it could be speedup. I guess better longer run
than break the test.

> msgstress01      180.44s
> fsx22            170.47s
> pty07            150.04s
> pty06            150.02s
> gf18             121.09s
> gf17             120.82s
> gf16             120.13s
> dirtyc0w_shmem   120.11s
> setsockopt07      76.47s

> In total, running the full ltp-lite suite currently takes ~1h20m, which is a bit
> long for CI or pre-merge validation pipelines.

What is ltp-lite? Is it your internal CI for LTP? bind06 is in cve and syscalls
runtest files.

> I'm wondering whether all these .runtime values are truly necessary to reproduce
> the intended issues (e.g., race conditions, fuzzing, sync timing issues).
> Or if we could:
>  - Set a lower threshold for .runtime on general-purpose stress/fuzz tests

I'd be careful for fuzzy sync.

>  - Introduce a runtime _policy_ for different environments (e.g., fast
> CI vs. full weekly runs)

Or filter out the long running tests if the purpose is just to test LTP itself
instead of the product?

> It might be beneficial to revisit the .runtime values of long-running tests and
> set a minimal yet effective duration that balances reproducibility with resource
> efficiency. This could help save time and free up test resources earlier.

Maybe Martin still have VM's which can trigger the problem to experiment, but
runtime probably differs across architectures and available resources (number of
CPU or memory).

Kind regards,
Petr

> Any thoughts/comments would be appreciated.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC] Reduce .runtime for Long-Running Tests ?

[Li Wang via ltp]

On Tue, May 27, 2025 at 7:10 PM Petr Vorel <pvorel@suse.cz> wrote:
>
> Hi Li,
>
> > Hello All,
>
> > After reviewing some LTP HTML test reports, I noticed that the ten tests
> > alone take nearly 20 minutes to complete. For example:
>
> > --------------------
> > bind06           300.15s
>
> This uses fuzzy sync, I wonder if it could be speedup. I guess better longer run
> than break the test.
>
> > msgstress01      180.44s
> > fsx22            170.47s
> > pty07            150.04s
> > pty06            150.02s
> > gf18             121.09s
> > gf17             120.82s
> > gf16             120.13s
> > dirtyc0w_shmem   120.11s
> > setsockopt07      76.47s
>
> > In total, running the full ltp-lite suite currently takes ~1h20m, which is a bit
> > long for CI or pre-merge validation pipelines.
>
> What is ltp-lite? Is it your internal CI for LTP? bind06 is in cve and syscalls
> runtest files.

Ah, that was combined by some general runtest and kicked out few
stressful tests:

Usually contains: math ipc syscalls mm sched nptl pty tracing fs ...

>
> > I'm wondering whether all these .runtime values are truly necessary to reproduce
> > the intended issues (e.g., race conditions, fuzzing, sync timing issues).
> > Or if we could:
> >  - Set a lower threshold for .runtime on general-purpose stress/fuzz tests
>
> I'd be careful for fuzzy sync.
>
> >  - Introduce a runtime _policy_ for different environments (e.g., fast
> > CI vs. full weekly runs)
>
> Or filter out the long running tests if the purpose is just to test LTP itself
> instead of the product?

Filter out some stressful test (we have done) can be acceptable, but I still
think some .runtime test is valuable to be executed in product verification.

That's why I started this email to get some inspiration.

>
> > It might be beneficial to revisit the .runtime values of long-running tests and
> > set a minimal yet effective duration that balances reproducibility with resource
> > efficiency. This could help save time and free up test resources earlier.
>
> Maybe Martin still have VM's which can trigger the problem to experiment, but
> runtime probably differs across architectures and available resources (number of
> CPU or memory).
>
> Kind regards,
> Petr
>
> > Any thoughts/comments would be appreciated.
>


-- 
Regards,
Li Wang


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC] Reduce .runtime for Long-Running Tests ?

[Li Wang via ltp]

On Tue, May 27, 2025 at 7:00 PM Li Wang <liwang@redhat.com> wrote:
>
> Hello All,
>
> After reviewing some LTP HTML test reports, I noticed that the ten tests
> alone take nearly 20 minutes to complete. For example:
>
> --------------------
> bind06           300.15s
> msgstress01      180.44s
> fsx22            170.47s
> pty07            150.04s
> pty06            150.02s
> gf18             121.09s
> gf17             120.82s
> gf16             120.13s
> dirtyc0w_shmem   120.11s
> setsockopt07      76.47s
>
> In total, running the full ltp-lite suite currently takes ~1h20m, which is a bit
> long for CI or pre-merge validation pipelines.

Forget to mention:

The 1h20min contains LTP compiling time. We now try to build rpm package
for our CI testing that would reduce some unnecessary time.

Apart from that, If we could do something to speed up the running time,
that would be great too.

>
> I'm wondering whether all these .runtime values are truly necessary to reproduce
> the intended issues (e.g., race conditions, fuzzing, sync timing issues).
> Or if we could:
>  - Set a lower threshold for .runtime on general-purpose stress/fuzz tests
>  - Introduce a runtime _policy_ for different environments (e.g., fast
> CI vs. full weekly runs)
>
> It might be beneficial to revisit the .runtime values of long-running tests and
> set a minimal yet effective duration that balances reproducibility with resource
> efficiency. This could help save time and free up test resources earlier.
>
> Any thoughts/comments would be appreciated.
>
> --
> Regards,
> Li Wang



-- 
Regards,
Li Wang


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC] Reduce .runtime for Long-Running Tests ?

[Cyril Hrubis]

Hi!
> After reviewing some LTP HTML test reports, I noticed that the ten tests
> alone take nearly 20 minutes to complete. For example:
> 
> --------------------
> bind06           300.15s
> msgstress01      180.44s
> fsx22            170.47s
> pty07            150.04s
> pty06            150.02s
> gf18             121.09s
> gf17             120.82s
> gf16             120.13s
> dirtyc0w_shmem   120.11s
> setsockopt07      76.47s
> 
> In total, running the full ltp-lite suite currently takes ~1h20m, which is a bit
> long for CI or pre-merge validation pipelines.
> 
> I'm wondering whether all these .runtime values are truly necessary to reproduce
> the intended issues (e.g., race conditions, fuzzing, sync timing issues).
> Or if we could:
>  - Set a lower threshold for .runtime on general-purpose stress/fuzz tests
>  - Introduce a runtime _policy_ for different environments (e.g., fast
> CI vs. full weekly runs)
> 
> It might be beneficial to revisit the .runtime values of long-running tests and
> set a minimal yet effective duration that balances reproducibility with resource
> efficiency. This could help save time and free up test resources earlier.
> 
> Any thoughts/comments would be appreciated.

This is exactly the reason why I want to have queries on metadata
available in kirk, e.g. "filter out all tests that have runtime >
$some_constant".

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC] Reduce .runtime for Long-Running Tests ?

[Andrea Cervesato via ltp]

On 5/27/25 13:19, Cyril Hrubis wrote:
> This is exactly the reason why I want to have queries on metadata
> available in kirk, e.g. "filter out all tests that have runtime >
> $some_constant".
It's already there and it's called "max_runtime". We only miss to 
convert all old tests into new LTP API:

$ ./kirk --framework=help
--framework option supports the following syntax:

<name>:<param1>=<value1>:<param2>=<value2>:..

Supported plugins: | ltp |

ltp configuration:
     root: LTP install folder
     max_runtime: filter out all tests above this time value

- Andrea


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC] Reduce .runtime for Long-Running Tests ?

[Cyril Hrubis]

Hi!
> > This is exactly the reason why I want to have queries on metadata
> > available in kirk, e.g. "filter out all tests that have runtime >
> > $some_constant".
> It's already there and it's called "max_runtime". We only miss to 
> convert all old tests into new LTP API:
> 
> $ ./kirk --framework=help
> --framework option supports the following syntax:
> 
> <name>:<param1>=<value1>:<param2>=<value2>:..
> 
> Supported plugins: | ltp |
> 
> ltp configuration:
>      root: LTP install folder
>      max_runtime: filter out all tests above this time value

It's much less than a general query language over the metadata I
envisioned, but at least this is a common usecase that is now covered...

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC] Reduce .runtime for Long-Running Tests ?

[Cyril Hrubis]

Hi!
> After reviewing some LTP HTML test reports, I noticed that the ten tests
> alone take nearly 20 minutes to complete. For example:
> 
> --------------------
> bind06           300.15s
> msgstress01      180.44s

Isn't this better after:

commit e3b85e50471b3892316a2b2c7c730e9dc8d9139e
Author: Ricardo B. Marlière <rbm@suse.com>
Date:   Wed May 21 05:31:12 2025 -0300

    syscalls/msgstress01: Set upper bound for num_messages


> fsx22            170.47s
> pty07            150.04s
> pty06            150.02s
> gf18             121.09s
> gf17             120.82s
> gf16             120.13s
> dirtyc0w_shmem   120.11s
> setsockopt07      76.47s

As Peter pointed out, most of these are fuzzy sync tests and the runtime
is callibrated in order to get reasonable chance of reproducing the
results. But indeed the runtime does not scale with the speed of the
machine and we are not setting the pair exec_loops in the tests that run
for so long, so I suppose that callibrating the number of loops for
these tests that does not make the bug less reproducible would make
things better.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC] Reduce .runtime for Long-Running Tests ?

[Martin Doucha]

On 28. 05. 25 15:35, Cyril Hrubis wrote:
> Hi!
>> After reviewing some LTP HTML test reports, I noticed that the ten tests
>> alone take nearly 20 minutes to complete. For example:
>>
>> --------------------
>> bind06           300.15s
>> msgstress01      180.44s
> 
> Isn't this better after:
> 
> commit e3b85e50471b3892316a2b2c7c730e9dc8d9139e
> Author: Ricardo B. Marlière <rbm@suse.com>
> Date:   Wed May 21 05:31:12 2025 -0300
> 
>      syscalls/msgstress01: Set upper bound for num_messages

This patch only fixes test failures due to very low max PID limit. It 
won't have any effect on machines which didn't get SAFE_FORK() failures 
before.

-- 
Martin Doucha   mdoucha@suse.cz
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC] Reduce .runtime for Long-Running Tests ?

[Li Wang via ltp]

On Wed, May 28, 2025 at 9:35 PM Cyril Hrubis <chrubis@suse.cz> wrote:
>
> Hi!
> > After reviewing some LTP HTML test reports, I noticed that the ten tests
> > alone take nearly 20 minutes to complete. For example:
> >
> > --------------------
> > bind06           300.15s
> > msgstress01      180.44s
>
> Isn't this better after:
>
> commit e3b85e50471b3892316a2b2c7c730e9dc8d9139e
> Author: Ricardo B. Marlière <rbm@suse.com>
> Date:   Wed May 21 05:31:12 2025 -0300
>
>     syscalls/msgstress01: Set upper bound for num_messages

This commit hasn't been backported to our CI ltp, so it hasn't gotten the count.
But later I will focus on that once we upgrade to the May release.

>
>
> > fsx22            170.47s
> > pty07            150.04s
> > pty06            150.02s
> > gf18             121.09s
> > gf17             120.82s
> > gf16             120.13s
> > dirtyc0w_shmem   120.11s
> > setsockopt07      76.47s
>
> As Peter pointed out, most of these are fuzzy sync tests and the runtime
> is callibrated in order to get reasonable chance of reproducing the
> results. But indeed the runtime does not scale with the speed of the
> machine and we are not setting the pair exec_loops in the tests that run

> for so long, so I suppose that callibrating the number of loops for
> these tests that does not make the bug less reproducible would make
> things better.

Agree, this is the direction that we need to focus on improving.

-- 
Regards,
Li Wang


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC] Reduce .runtime for Long-Running Tests ?

[Li Wang via ltp]

Hi All,

After a round of experiments and internal discussions (thanks to
Ian Wien for sharing his thoughts with me), we think that making
LTP_RUNTIME_MUL support floating point numbers [0, 1] may
a possible way to reduce the .runtime values set in tests.

For example, setting LTP_RUNTIME_MUL to 0.1 can obviously
reduce the test time of 600 seconds to 60 seconds.

One may think that reducing the .runtime value in aproduction
environment is potentially risky, and to some extent the answer
is yes.

But looking back, LTP is triggered very frequently in CI and various
production flows, so to compensate for this loss, we can use floating
point LTP_RUNTIME_MUL only in designated quick CI, instead of
using it in daily tests. This will help cover more scenarios.

From our CI report, use 0.1 in runtime_mul find a few failures in the round
down problem with nice05.c (.runtime = 3), this is a defect of
multply_runtime().
Also, another preadv203 possible failure related this. But they are tiny
issues. And the rest .runtime tests so far no obvious problem on that.

So I would like to start the work from this point to reduce execution time.

Regarding calibrating the number of loops, I think that is another good
direction to work, but may need more time to evaluate. I think that's a
long-term goal.


On Thu, May 29, 2025 at 9:53 AM Li Wang <liwang@redhat.com> wrote:

> On Wed, May 28, 2025 at 9:35 PM Cyril Hrubis <chrubis@suse.cz> wrote:
> >
> > Hi!
> > > After reviewing some LTP HTML test reports, I noticed that the ten
> tests
> > > alone take nearly 20 minutes to complete. For example:
> > >
> > > --------------------
> > > bind06           300.15s
> > > msgstress01      180.44s
> >
> > Isn't this better after:
> >
> > commit e3b85e50471b3892316a2b2c7c730e9dc8d9139e
> > Author: Ricardo B. Marlière <rbm@suse.com>
> > Date:   Wed May 21 05:31:12 2025 -0300
> >
> >     syscalls/msgstress01: Set upper bound for num_messages
>
> This commit hasn't been backported to our CI ltp, so it hasn't gotten the
> count.
> But later I will focus on that once we upgrade to the May release.
>
> >
> >
> > > fsx22            170.47s
> > > pty07            150.04s
> > > pty06            150.02s
> > > gf18             121.09s
> > > gf17             120.82s
> > > gf16             120.13s
> > > dirtyc0w_shmem   120.11s
> > > setsockopt07      76.47s
> >
> > As Peter pointed out, most of these are fuzzy sync tests and the runtime
> > is callibrated in order to get reasonable chance of reproducing the
> > results. But indeed the runtime does not scale with the speed of the
> > machine and we are not setting the pair exec_loops in the tests that run
>
> > for so long, so I suppose that callibrating the number of loops for
> > these tests that does not make the bug less reproducible would make
> > things better.
>
> Agree, this is the direction that we need to focus on improving.
>
> --
> Regards,
> Li Wang
>


-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC] Reduce .runtime for Long-Running Tests ?

[Cyril Hrubis]

Hi!
> After a round of experiments and internal discussions (thanks to
> Ian Wien for sharing his thoughts with me), we think that making
> LTP_RUNTIME_MUL support floating point numbers [0, 1] may
> a possible way to reduce the .runtime values set in tests.
> 
> For example, setting LTP_RUNTIME_MUL to 0.1 can obviously
> reduce the test time of 600 seconds to 60 seconds.
> 
> One may think that reducing the .runtime value in aproduction
> environment is potentially risky, and to some extent the answer
> is yes.
> 
> But looking back, LTP is triggered very frequently in CI and various
> production flows, so to compensate for this loss, we can use floating
> point LTP_RUNTIME_MUL only in designated quick CI, instead of
> using it in daily tests. This will help cover more scenarios.

Having reduced runtime for CI makes sense, as you said you are making up
for the reduction by the number of testruns. It may not work well for
fuzzy sync though where we depend on having good enough sampling period.

Also limiting the smallest multiplier to be 0.1 or so does make sense. I
assume that if we set it to 0.01 or smaller most tests with runtime
woudn't even execute.

> From our CI report, use 0.1 in runtime_mul find a few failures in the round
> down problem with nice05.c (.runtime = 3), this is a defect of
> multply_runtime().
> Also, another preadv203 possible failure related this. But they are tiny
> issues. And the rest .runtime tests so far no obvious problem on that.

I guess that we need to make sure that the runtime stays positive. So
maybe we need something as:

runtime = MAX(2, tst_test->runtime * runtime_mul)

> So I would like to start the work from this point to reduce execution time.

Sounds good to me.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC] Reduce .runtime for Long-Running Tests ?

[Li Wang via ltp]

On Fri, Jun 20, 2025 at 5:03 PM Cyril Hrubis <chrubis@suse.cz> wrote:

> Hi!
> > After a round of experiments and internal discussions (thanks to
> > Ian Wien for sharing his thoughts with me), we think that making
> > LTP_RUNTIME_MUL support floating point numbers [0, 1] may
> > a possible way to reduce the .runtime values set in tests.
> >
> > For example, setting LTP_RUNTIME_MUL to 0.1 can obviously
> > reduce the test time of 600 seconds to 60 seconds.
> >
> > One may think that reducing the .runtime value in aproduction
> > environment is potentially risky, and to some extent the answer
> > is yes.
> >
> > But looking back, LTP is triggered very frequently in CI and various
> > production flows, so to compensate for this loss, we can use floating
> > point LTP_RUNTIME_MUL only in designated quick CI, instead of
> > using it in daily tests. This will help cover more scenarios.
>
>


> Having reduced runtime for CI makes sense, as you said you are making up
> for the reduction by the number of testruns. It may not work well for
> fuzzy sync though where we depend on having good enough sampling period.
>

Against this requirement, maybe we can introduce a "switch button" to close
runtime_multiply (0.1] take effective in fuzzy sync sampling tests, even for
some other tests thatrely on tst_remaind_runtime to control the process
(e..g preadv203) as well

With this switch, we can precisely control the specified test not get
impacted
by the value of LTP_RUNTIME_MUL(whatever <1 or >1).



>
> Also limiting the smallest multiplier to be 0.1 or so does make sense. I
> assume that if we set it to 0.01 or smaller most tests with runtime
> woudn't even execute.
>
> > From our CI report, use 0.1 in runtime_mul find a few failures in the
> round
> > down problem with nice05.c (.runtime = 3), this is a defect of
> > multply_runtime().
> > Also, another preadv203 possible failure related this. But they are tiny
> > issues. And the rest .runtime tests so far no obvious problem on that.
>
> I guess that we need to make sure that the runtime stays positive. So
> maybe we need something as:
>
> runtime = MAX(2, tst_test->runtime * runtime_mul)
>
> > So I would like to start the work from this point to reduce execution
> time.
>
> Sounds good to me.
>
> --
> Cyril Hrubis
> chrubis@suse.cz
>
>

-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp