* [LTP] LTP in valgrind :) @ 2025-04-29 5:22 Petr Vorel 2025-04-29 6:59 ` Andrea Cervesato via ltp 2025-05-02 9:49 ` Cyril Hrubis 0 siblings, 2 replies; 17+ messages in thread From: Petr Vorel @ 2025-04-29 5:22 UTC (permalink / raw) To: ltp Hi all, in case you haven't read the news in LWN, LTP is now part of valgrind [1]: The Linux Test Project (ltp) is integrated in the testsuite try 'make ltpchecks' (this will take a while and will point out various missing syscalls and valgrind crashes!) FYI script they use [2]. They consider some test long running [3]. Kind regards, Petr [1] https://www.mail-archive.com/valgrind-users@lists.sourceforge.net/msg07691.html [2] https://sourceware.org/git/?p=valgrind.git;a=blob;f=auxprogs/ltp-tester.sh;h=000cfaa7f336f8641f464d107a363af24f607614;hb=HEAD [3] https://sourceware.org/git/?p=valgrind.git;a=blob;f=auxprogs/ltp-error-patterns.txt;h=ee5048bf42b652258fb469a632fb2d16afb72c38;hb=HEAD -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [LTP] LTP in valgrind :) 2025-04-29 5:22 [LTP] LTP in valgrind :) Petr Vorel @ 2025-04-29 6:59 ` Andrea Cervesato via ltp 2025-05-02 9:49 ` Cyril Hrubis 1 sibling, 0 replies; 17+ messages in thread From: Andrea Cervesato via ltp @ 2025-04-29 6:59 UTC (permalink / raw) To: Petr Vorel, ltp Hi Petr, what a nice surprise :-) I guess polishing and fixing LTP somehow helped. Kind regards, Andrea Cervesato On 4/29/25 07:22, Petr Vorel wrote: > Hi all, > > in case you haven't read the news in LWN, LTP is now part of valgrind [1]: > > The Linux Test Project (ltp) is integrated in the testsuite try > 'make ltpchecks' (this will take a while and will point out various > missing syscalls and valgrind crashes!) > > FYI script they use [2]. They consider some test long running [3]. > > Kind regards, > Petr > > [1] https://www.mail-archive.com/valgrind-users@lists.sourceforge.net/msg07691.html > [2] https://sourceware.org/git/?p=valgrind.git;a=blob;f=auxprogs/ltp-tester.sh;h=000cfaa7f336f8641f464d107a363af24f607614;hb=HEAD > [3] https://sourceware.org/git/?p=valgrind.git;a=blob;f=auxprogs/ltp-error-patterns.txt;h=ee5048bf42b652258fb469a632fb2d16afb72c38;hb=HEAD > -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [LTP] LTP in valgrind :) 2025-04-29 5:22 [LTP] LTP in valgrind :) Petr Vorel 2025-04-29 6:59 ` Andrea Cervesato via ltp @ 2025-05-02 9:49 ` Cyril Hrubis 2025-05-02 10:53 ` Martin Doucha 2025-05-05 19:50 ` Petr Vorel 1 sibling, 2 replies; 17+ messages in thread From: Cyril Hrubis @ 2025-05-02 9:49 UTC (permalink / raw) To: Petr Vorel; +Cc: ltp Hi! Awesome :-) It also opens some interesting questions, i.e. how do we make comparing results from two different tests easier. Currently they grep the test results for a summary, but maybe we can do better. -- Cyril Hrubis chrubis@suse.cz -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [LTP] LTP in valgrind :) 2025-05-02 9:49 ` Cyril Hrubis @ 2025-05-02 10:53 ` Martin Doucha 2025-05-05 8:57 ` Cyril Hrubis 2025-05-05 19:50 ` Petr Vorel 1 sibling, 1 reply; 17+ messages in thread From: Martin Doucha @ 2025-05-02 10:53 UTC (permalink / raw) To: Cyril Hrubis, Petr Vorel; +Cc: ltp On 02. 05. 25 11:49, Cyril Hrubis wrote: > Hi! > Awesome :-) > > It also opens some interesting questions, i.e. how do we make comparing > results from two different tests easier. Currently they grep the test > results for a summary, but maybe we can do better. One option is to extract all TPASS/TFAIL/TWARN/TBROK/TCONF messages, discard any message contents past the file:line header and then compare whether the sanitized output is identical. That'll take care of random values in the output while ensuring that the test went through the same code paths as before. We could provide a sanitizer script for that. -- Martin Doucha mdoucha@suse.cz SW Quality Engineer SUSE LINUX, s.r.o. CORSO IIa Krizikova 148/34 186 00 Prague 8 Czech Republic -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [LTP] LTP in valgrind :) 2025-05-02 10:53 ` Martin Doucha @ 2025-05-05 8:57 ` Cyril Hrubis 2025-05-05 19:52 ` Petr Vorel 2025-05-06 8:05 ` Petr Vorel 0 siblings, 2 replies; 17+ messages in thread From: Cyril Hrubis @ 2025-05-05 8:57 UTC (permalink / raw) To: Martin Doucha; +Cc: ltp Hi! > > It also opens some interesting questions, i.e. how do we make comparing > > results from two different tests easier. Currently they grep the test > > results for a summary, but maybe we can do better. > > One option is to extract all TPASS/TFAIL/TWARN/TBROK/TCONF messages, > discard any message contents past the file:line header and then compare > whether the sanitized output is identical. That'll take care of random > values in the output while ensuring that the test went through the same > code paths as before. We could provide a sanitizer script for that. Maybe we can even add an option to the test library to supress the messages in output, that would be fairly simple. -- Cyril Hrubis chrubis@suse.cz -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [LTP] LTP in valgrind :) 2025-05-05 8:57 ` Cyril Hrubis @ 2025-05-05 19:52 ` Petr Vorel 2025-05-06 8:05 ` Petr Vorel 1 sibling, 0 replies; 17+ messages in thread From: Petr Vorel @ 2025-05-05 19:52 UTC (permalink / raw) To: Cyril Hrubis; +Cc: ltp > Hi! > > > It also opens some interesting questions, i.e. how do we make comparing > > > results from two different tests easier. Currently they grep the test > > > results for a summary, but maybe we can do better. > > One option is to extract all TPASS/TFAIL/TWARN/TBROK/TCONF messages, > > discard any message contents past the file:line header and then compare > > whether the sanitized output is identical. That'll take care of random > > values in the output while ensuring that the test went through the same > > code paths as before. We could provide a sanitizer script for that. > Maybe we can even add an option to the test library to supress the > messages in output, that would be fairly simple. +1 And Cc the authors :). Kind regards, Petr -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [LTP] LTP in valgrind :) 2025-05-05 8:57 ` Cyril Hrubis 2025-05-05 19:52 ` Petr Vorel @ 2025-05-06 8:05 ` Petr Vorel 2025-05-07 15:11 ` Martin Cermak via ltp 2025-05-08 16:40 ` Mark Wielaard 1 sibling, 2 replies; 17+ messages in thread From: Petr Vorel @ 2025-05-06 8:05 UTC (permalink / raw) To: Cyril Hrubis; +Cc: Mark Wielaard, ltp, Martin Cermak Hi all, [ Cc Martin and Mark from Valgrind project ] > Hi! > > > It also opens some interesting questions, i.e. how do we make comparing > > > results from two different tests easier. Currently they grep the test > > > results for a summary, but maybe we can do better. > > One option is to extract all TPASS/TFAIL/TWARN/TBROK/TCONF messages, > > discard any message contents past the file:line header and then compare > > whether the sanitized output is identical. That'll take care of random > > values in the output while ensuring that the test went through the same > > code paths as before. We could provide a sanitizer script for that. > Maybe we can even add an option to the test library to supress the > messages in output, that would be fairly simple. @Martin @Mark: feel free to comment what we can do for you :). Whole thread: https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#t Kind regards, Petr -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [LTP] LTP in valgrind :) 2025-05-06 8:05 ` Petr Vorel @ 2025-05-07 15:11 ` Martin Cermak via ltp 2025-05-07 15:29 ` Cyril Hrubis 2025-05-08 16:40 ` Mark Wielaard 1 sibling, 1 reply; 17+ messages in thread From: Martin Cermak via ltp @ 2025-05-07 15:11 UTC (permalink / raw) To: Petr Vorel; +Cc: Mark Wielaard, ltp Hello guys, On Tue 2025-05-06 10:05 , Petr Vorel wrote: > Hi all, > > [ Cc Martin and Mark from Valgrind project ] > > > Hi! > > > > It also opens some interesting questions, i.e. how do we make comparing > > > > results from two different tests easier. Currently they grep the test > > > > results for a summary, but maybe we can do better. > > > > One option is to extract all TPASS/TFAIL/TWARN/TBROK/TCONF messages, > > > discard any message contents past the file:line header and then compare > > > whether the sanitized output is identical. That'll take care of random > > > values in the output while ensuring that the test went through the same > > > code paths as before. We could provide a sanitizer script for that. > > > Maybe we can even add an option to the test library to supress the > > messages in output, that would be fairly simple. > > @Martin @Mark: feel free to comment what we can do for you :). > Whole thread: > https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#t Nice to see you noticed! Primarily we're trying to identify valgrind gaps and bugs. For our use-case it's good to have reproducible test runs in a sense of identical test results for repeated test runs. We got pretty close to this by only taking into account the testcase summaries at the very end of the test logs. Any steps to get closer to reproducible test runs on the LTP side are interesting for us, and we'll try to reflect those! Cheers, Martin -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [LTP] LTP in valgrind :) 2025-05-07 15:11 ` Martin Cermak via ltp @ 2025-05-07 15:29 ` Cyril Hrubis 2025-05-08 16:07 ` Mark Wielaard 0 siblings, 1 reply; 17+ messages in thread From: Cyril Hrubis @ 2025-05-07 15:29 UTC (permalink / raw) To: Martin Cermak; +Cc: Mark Wielaard, ltp Hi! > > > > > It also opens some interesting questions, i.e. how do we make comparing > > > > > results from two different tests easier. Currently they grep the test > > > > > results for a summary, but maybe we can do better. > > > > > > One option is to extract all TPASS/TFAIL/TWARN/TBROK/TCONF messages, > > > > discard any message contents past the file:line header and then compare > > > > whether the sanitized output is identical. That'll take care of random > > > > values in the output while ensuring that the test went through the same > > > > code paths as before. We could provide a sanitizer script for that. > > > > > Maybe we can even add an option to the test library to supress the > > > messages in output, that would be fairly simple. > > > > @Martin @Mark: feel free to comment what we can do for you :). > > Whole thread: > > https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#t > > Nice to see you noticed! Primarily we're trying to identify > valgrind gaps and bugs. For our use-case it's good to have > reproducible test runs in a sense of identical test results for > repeated test runs. We got pretty close to this by only taking > into account the testcase summaries at the very end of the test > logs. Any steps to get closer to reproducible test runs on the > LTP side are interesting for us, and we'll try to reflect those! I've proposed a simple solution that would cut the message from tests only to contain the line numbers and results in: https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#m98317262b25ede62704b005fcefc63c62198bb90 If that works for you we can get this in LTP before the next release. -- Cyril Hrubis chrubis@suse.cz -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [LTP] LTP in valgrind :) 2025-05-07 15:29 ` Cyril Hrubis @ 2025-05-08 16:07 ` Mark Wielaard 2025-05-09 8:24 ` Cyril Hrubis 2025-05-09 14:16 ` Bird, Tim 0 siblings, 2 replies; 17+ messages in thread From: Mark Wielaard @ 2025-05-08 16:07 UTC (permalink / raw) To: Cyril Hrubis, Martin Cermak; +Cc: ltp Hi Cyril, Hi ltp hackers, On Wed, 2025-05-07 at 17:29 +0200, Cyril Hrubis wrote: > > > > > > > > > Maybe we can even add an option to the test library to supress the > > > > messages in output, that would be fairly simple. > > > > > > @Martin @Mark: feel free to comment what we can do for you :). > > > Whole thread: > > > https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#t > > > > Nice to see you noticed! Primarily we're trying to identify > > valgrind gaps and bugs. For our use-case it's good to have > > reproducible test runs in a sense of identical test results for > > repeated test runs. We got pretty close to this by only taking > > into account the testcase summaries at the very end of the test > > logs. Any steps to get closer to reproducible test runs on the > > LTP side are interesting for us, and we'll try to reflect those! > > I've proposed a simple solution that would cut the message from tests > only to contain the line numbers and results in: > > https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#m98317262b25ede62704b005fcefc63c62198bb90 > > If that works for you we can get this in LTP before the next release. > That looks pretty interesting, I just tried it and it turns: $ ./testcases/kernel/syscalls/dup/dup01 tst_tmpdir.c:316: TINFO: Using /tmp/LTP_dupbmNNGn as tmpdir (tmpfs filesystem) tst_test.c:1907: TINFO: LTP version: VALGRIND_3_25_0-10-g0ce068434ec3 tst_test.c:1911: TINFO: Tested kernel: 6.13.11-200.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 10 19:02:09 UTC 2025 x86_64 tst_kconfig.c:88: TINFO: Parsing kernel config '/lib/modules/6.13.11-200.fc41.x86_64/build/.config' tst_test.c:1729: TINFO: Overall timeout per run is 0h 00m 30s dup01.c:24: TPASS: dup(fd) returned fd 4 dup01.c:27: TPASS: buf1.st_ino == buf2.st_ino (6921515) Summary: passed 2 failed 0 broken 0 skipped 0 warnings 0 into: $ LTP_REPRODUCIBLE_OUTPUT=1 ./testcases/kernel/syscalls/dup/dup01 tst_tmpdir.c:316: tst_test.c:1907: tst_test.c:1911: tst_kconfig.c:88: tst_test.c:1729: dup01.c:24: dup01.c:27: Summary: passed 2 failed 0 broken 0 skipped 0 warnings 0 So that still shows the "code flow" and the test results Summary (all output goes to stderr). I think this would indeed be useful to compare if a testcase runs the same "native" and under valgrind. Thanks, Mark -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [LTP] LTP in valgrind :) 2025-05-08 16:07 ` Mark Wielaard @ 2025-05-09 8:24 ` Cyril Hrubis 2025-05-09 14:16 ` Bird, Tim 1 sibling, 0 replies; 17+ messages in thread From: Cyril Hrubis @ 2025-05-09 8:24 UTC (permalink / raw) To: Mark Wielaard; +Cc: Martin Cermak, ltp Hi! > That looks pretty interesting, I just tried it and it turns: > > $ ./testcases/kernel/syscalls/dup/dup01 > tst_tmpdir.c:316: TINFO: Using /tmp/LTP_dupbmNNGn as tmpdir (tmpfs filesystem) > tst_test.c:1907: TINFO: LTP version: VALGRIND_3_25_0-10-g0ce068434ec3 > tst_test.c:1911: TINFO: Tested kernel: 6.13.11-200.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 10 19:02:09 UTC 2025 x86_64 > tst_kconfig.c:88: TINFO: Parsing kernel config '/lib/modules/6.13.11-200.fc41.x86_64/build/.config' > tst_test.c:1729: TINFO: Overall timeout per run is 0h 00m 30s > dup01.c:24: TPASS: dup(fd) returned fd 4 > dup01.c:27: TPASS: buf1.st_ino == buf2.st_ino (6921515) > > Summary: > passed 2 > failed 0 > broken 0 > skipped 0 > warnings 0 > > into: > > $ LTP_REPRODUCIBLE_OUTPUT=1 ./testcases/kernel/syscalls/dup/dup01 > tst_tmpdir.c:316: > tst_test.c:1907: > tst_test.c:1911: > tst_kconfig.c:88: > tst_test.c:1729: > dup01.c:24: > dup01.c:27: > > Summary: > passed 2 > failed 0 > broken 0 > skipped 0 > warnings 0 > > So that still shows the "code flow" and the test results Summary (all > output goes to stderr). I think this would indeed be useful to compare > if a testcase runs the same "native" and under valgrind. I will send a proper LTP patch. -- Cyril Hrubis chrubis@suse.cz -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [LTP] LTP in valgrind :) 2025-05-08 16:07 ` Mark Wielaard 2025-05-09 8:24 ` Cyril Hrubis @ 2025-05-09 14:16 ` Bird, Tim 2025-05-12 8:09 ` Cyril Hrubis 1 sibling, 1 reply; 17+ messages in thread From: Bird, Tim @ 2025-05-09 14:16 UTC (permalink / raw) To: Mark Wielaard, Cyril Hrubis, Martin Cermak; +Cc: ltp@lists.linux.it > -----Original Message----- > From: ltp <ltp-bounces+tim.bird=sony.com@lists.linux.it> On Behalf Of Mark Wielaard > Hi Cyril, Hi ltp hackers, > > On Wed, 2025-05-07 at 17:29 +0200, Cyril Hrubis wrote: > > > > > > > > > > > Maybe we can even add an option to the test library to supress the > > > > > messages in output, that would be fairly simple. > > > > > > > > @Martin @Mark: feel free to comment what we can do for you :). > > > > Whole thread: > > > > https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#t > > > > > > Nice to see you noticed! Primarily we're trying to identify > > > valgrind gaps and bugs. For our use-case it's good to have > > > reproducible test runs in a sense of identical test results for > > > repeated test runs. We got pretty close to this by only taking > > > into account the testcase summaries at the very end of the test > > > logs. Any steps to get closer to reproducible test runs on the > > > LTP side are interesting for us, and we'll try to reflect those! > > > > I've proposed a simple solution that would cut the message from tests > > only to contain the line numbers and results in: > > > > https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#m98317262b25ede62704b005fcefc63c62198bb90 > > > > If that works for you we can get this in LTP before the next release. > > > > That looks pretty interesting, I just tried it and it turns: > > $ ./testcases/kernel/syscalls/dup/dup01 > tst_tmpdir.c:316: TINFO: Using /tmp/LTP_dupbmNNGn as tmpdir (tmpfs filesystem) > tst_test.c:1907: TINFO: LTP version: VALGRIND_3_25_0-10-g0ce068434ec3 > tst_test.c:1911: TINFO: Tested kernel: 6.13.11-200.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 10 19:02:09 UTC 2025 x86_64 > tst_kconfig.c:88: TINFO: Parsing kernel config '/lib/modules/6.13.11-200.fc41.x86_64/build/.config' > tst_test.c:1729: TINFO: Overall timeout per run is 0h 00m 30s > dup01.c:24: TPASS: dup(fd) returned fd 4 > dup01.c:27: TPASS: buf1.st_ino == buf2.st_ino (6921515) > > Summary: > passed 2 > failed 0 > broken 0 > skipped 0 > warnings 0 > > into: > > $ LTP_REPRODUCIBLE_OUTPUT=1 ./testcases/kernel/syscalls/dup/dup01 > tst_tmpdir.c:316: > tst_test.c:1907: > tst_test.c:1911: > tst_kconfig.c:88: > tst_test.c:1729: > dup01.c:24: > dup01.c:27: Don't you want to keep the status or result (TINFO and TPASS)? This shouldn't change from run-to-run unless the test is flaky. -- Tim > Summary: > passed 2 > failed 0 > broken 0 > skipped 0 > warnings 0 > > So that still shows the "code flow" and the test results Summary (all > output goes to stderr). I think this would indeed be useful to compare > if a testcase runs the same "native" and under valgrind. > > Thanks, > > Mark > > -- > Mailing list info: https://lists.linux.it/listinfo/ltp -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [LTP] LTP in valgrind :) 2025-05-09 14:16 ` Bird, Tim @ 2025-05-12 8:09 ` Cyril Hrubis 0 siblings, 0 replies; 17+ messages in thread From: Cyril Hrubis @ 2025-05-12 8:09 UTC (permalink / raw) To: Bird, Tim; +Cc: Mark Wielaard, Martin Cermak, ltp@lists.linux.it Hi! > Don't you want to keep the status or result (TINFO and TPASS)? > > This shouldn't change from run-to-run unless the test is flaky. Already addressed in v2 patch: http://patchwork.ozlabs.org/project/ltp/patch/20250509092813.12860-1-chrubis@suse.cz/ -- Cyril Hrubis chrubis@suse.cz -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [LTP] LTP in valgrind :) 2025-05-06 8:05 ` Petr Vorel 2025-05-07 15:11 ` Martin Cermak via ltp @ 2025-05-08 16:40 ` Mark Wielaard 1 sibling, 0 replies; 17+ messages in thread From: Mark Wielaard @ 2025-05-08 16:40 UTC (permalink / raw) To: Petr Vorel, Cyril Hrubis; +Cc: ltp, Martin Cermak Hi Petr, Hi ltp hackers, On Tue, 2025-05-06 at 10:05 +0200, Petr Vorel wrote: > > > > It also opens some interesting questions, i.e. how do we make comparing > > > > results from two different tests easier. Currently they grep the test > > > > results for a summary, but maybe we can do better. > > > > One option is to extract all TPASS/TFAIL/TWARN/TBROK/TCONF messages, > > > discard any message contents past the file:line header and then compare > > > whether the sanitized output is identical. That'll take care of random > > > values in the output while ensuring that the test went through the same > > > code paths as before. We could provide a sanitizer script for that. > > > Maybe we can even add an option to the test library to supress the > > messages in output, that would be fairly simple. > > @Martin @Mark: feel free to comment what we can do for you :). > Whole thread: > https://lore.kernel.org/ltp/20250505195003.GB137650@pevik/T/#t Thanks. Something small, but if you could CC valgrind-developers@lists.sourceforge.net on new ltp releases that would be great. I saw a question at the top of that thread about "They consider some test long running". In general that is not about the actual ltp tests (we currently only run the testcases/kernel/syscalls ones), but because valgrind: a) is basically a giant Just In Time compiler, which instruments all code, and might slow down your code 20x (yeah, we know). b) serializes all threads, so only one thread runs at a time, all parallel code is executed as if it was serial code. c) memcheck keeps track of all memory to see if values are "valid" (have a known value written to them). This cost about 25% more memory than the program itself uses (and might actually double the amount). So for now we have the following exclude file: https://sourceware.org/cgit/valgrind/tree/auxprogs/ltp-excludes.txt bind06 epoll-ltp fork14 futex_cmp_requeue01 futex_cmp_requeue02 inotify09 msgstress01 pidfd_send_signal01 pidfd_send_signal02 pidfd_send_signal03 sendmsg03 setsockopt06 setsockopt07 signal05 signal06 timerfd_settime02 We haven't yet analyzed them to see what exactly makes them so slow (under valgrind). It was more important to get the ltpchecks run on all our CI systems: https://builder.sourceware.org/buildbot/#/builders?tags=valgrind Some of those systems only have 8GB of memory or are fairly slow (specifically the arm32 and riscv builders). The pidfd_send_signal tests are excluded because they seemed to kill the container they were running inside. Again, this could be a valgrind bug. valgrind doesn't implement a proper syscall wrapper for pidfd_send_signal so it might be possible we are just killing a random process... Cheers, Mark -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [LTP] LTP in valgrind :) 2025-05-02 9:49 ` Cyril Hrubis 2025-05-02 10:53 ` Martin Doucha @ 2025-05-05 19:50 ` Petr Vorel 2025-05-06 8:03 ` Cyril Hrubis 1 sibling, 1 reply; 17+ messages in thread From: Petr Vorel @ 2025-05-05 19:50 UTC (permalink / raw) To: Cyril Hrubis; +Cc: ltp > Hi! > Awesome :-) > It also opens some interesting questions, i.e. how do we make comparing > results from two different tests easier. Currently they grep the test > results for a summary, but maybe we can do better. Kirk and compare json? Kind regards, Petr -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [LTP] LTP in valgrind :) 2025-05-05 19:50 ` Petr Vorel @ 2025-05-06 8:03 ` Cyril Hrubis 2025-05-06 8:25 ` Petr Vorel 0 siblings, 1 reply; 17+ messages in thread From: Cyril Hrubis @ 2025-05-06 8:03 UTC (permalink / raw) To: Petr Vorel; +Cc: ltp Hi! > > It also opens some interesting questions, i.e. how do we make comparing > > results from two different tests easier. Currently they grep the test > > results for a summary, but maybe we can do better. > > Kirk and compare json? That does sound like an overkill for something that is basically: diff --git a/lib/tst_test.c b/lib/tst_test.c index 694861d95..29d9fc1d5 100644 --- a/lib/tst_test.c +++ b/lib/tst_test.c @@ -64,6 +64,7 @@ static int mntpoint_mounted; static int ovl_mounted; static struct timespec tst_start_time; /* valid only for test pid */ static int tdebug; +static int reproducible_output; struct results { int passed; @@ -299,6 +300,9 @@ static void print_result(const char *file, const int lineno, int ttype, str += ret; size -= ret; + if (reproducible_output) + goto print; + if (tst_color_enabled(STDERR_FILENO)) ret = snprintf(str, size, "%s%s: %s", tst_ttype2color(ttype), res, ANSI_COLOR_RESET); @@ -324,6 +328,7 @@ static void print_result(const char *file, const int lineno, int ttype, "Next message is too long and truncated:"); } +print: snprintf(str, size, "\n"); /* we might be called from signal handler, so use write() */ @@ -1273,6 +1278,8 @@ static void do_setup(int argc, char *argv[]) if (tst_test->tconf_msg) tst_brk(TCONF, "%s", tst_test->tconf_msg); + reproducible_output = !!getenv("LTP_REPRODUCIBLE_OUTPUT"); + assert_test_fn(); TCID = tid = get_tid(argv); -- Cyril Hrubis chrubis@suse.cz -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [LTP] LTP in valgrind :) 2025-05-06 8:03 ` Cyril Hrubis @ 2025-05-06 8:25 ` Petr Vorel 0 siblings, 0 replies; 17+ messages in thread From: Petr Vorel @ 2025-05-06 8:25 UTC (permalink / raw) To: Cyril Hrubis; +Cc: Mark Wielaard, Martin Cermak, ltp Hi Cyril, > Hi! > > > It also opens some interesting questions, i.e. how do we make comparing > > > results from two different tests easier. Currently they grep the test > > > results for a summary, but maybe we can do better. > > Kirk and compare json? > That does sound like an overkill for something that is basically: Agree. I sometimes think JSON output would be useful to be even in core LTP (not requiring kirk), but that's something different than reproducible output. The patch above LGTM (if you send patch please remember to update doc/users/setup_tests.rst). Kind regards, Petr > diff --git a/lib/tst_test.c b/lib/tst_test.c > index 694861d95..29d9fc1d5 100644 > --- a/lib/tst_test.c > +++ b/lib/tst_test.c > @@ -64,6 +64,7 @@ static int mntpoint_mounted; > static int ovl_mounted; > static struct timespec tst_start_time; /* valid only for test pid */ > static int tdebug; > +static int reproducible_output; > struct results { > int passed; > @@ -299,6 +300,9 @@ static void print_result(const char *file, const int lineno, int ttype, > str += ret; > size -= ret; > + if (reproducible_output) > + goto print; > + > if (tst_color_enabled(STDERR_FILENO)) > ret = snprintf(str, size, "%s%s: %s", tst_ttype2color(ttype), > res, ANSI_COLOR_RESET); > @@ -324,6 +328,7 @@ static void print_result(const char *file, const int lineno, int ttype, > "Next message is too long and truncated:"); > } > +print: > snprintf(str, size, "\n"); > /* we might be called from signal handler, so use write() */ > @@ -1273,6 +1278,8 @@ static void do_setup(int argc, char *argv[]) > if (tst_test->tconf_msg) > tst_brk(TCONF, "%s", tst_test->tconf_msg); > + reproducible_output = !!getenv("LTP_REPRODUCIBLE_OUTPUT"); > + > assert_test_fn(); > TCID = tid = get_tid(argv); -- Mailing list info: https://lists.linux.it/listinfo/ltp ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2025-05-12 8:08 UTC | newest] Thread overview: 17+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-04-29 5:22 [LTP] LTP in valgrind :) Petr Vorel 2025-04-29 6:59 ` Andrea Cervesato via ltp 2025-05-02 9:49 ` Cyril Hrubis 2025-05-02 10:53 ` Martin Doucha 2025-05-05 8:57 ` Cyril Hrubis 2025-05-05 19:52 ` Petr Vorel 2025-05-06 8:05 ` Petr Vorel 2025-05-07 15:11 ` Martin Cermak via ltp 2025-05-07 15:29 ` Cyril Hrubis 2025-05-08 16:07 ` Mark Wielaard 2025-05-09 8:24 ` Cyril Hrubis 2025-05-09 14:16 ` Bird, Tim 2025-05-12 8:09 ` Cyril Hrubis 2025-05-08 16:40 ` Mark Wielaard 2025-05-05 19:50 ` Petr Vorel 2025-05-06 8:03 ` Cyril Hrubis 2025-05-06 8:25 ` Petr Vorel
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.