[LTP] Question about log format.

[LTP] Question about log format.

[Xinkuan Yan]

Hello list,

I'm trying to do a concise parsing of the LTP report.

I don't have much experience and I'm not familiar with LTP, so I'm going to
take #math as an example, for it is not complex.

Run test command:

./ltp –f math –q

I have three questions now,  taking the one-line output result below as an
example:

float_iperb 1 TPASS: Test passed

1, It is clear that TPASS means it passed, but what is the number 1
standing for.

2, Why this line showed up twice? ( PS: Between these two lines are many
TINFO.)

3, Where to check those out put standards, taking yocto package test as a
small comprehension, that standard was ( PASS|FAIL|SKIP): Test info. How is
it like in LTP?

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] Question about log format.

[Cyril Hrubis]

Hi!
> I'm trying to do a concise parsing of the LTP report.
> 
> I don't have much experience and I'm not familiar with LTP, so I'm going to
> take #math as an example, for it is not complex.
>
> Run test command:
> 
> ./ltp –f math –q
> 
> I have three questions now,  taking the one-line output result below as an
> example:
> 
> float_iperb 1 TPASS: Test passed
> 
> 1, It is clear that TPASS means it passed, but what is the number 1
> standing for.
> 
> 2, Why this line showed up twice? ( PS: Between these two lines are many
> TINFO.)

The math tests are prime example how not to write tests. Please do not
take them as an example of anything else. Usually the most clean and up
to date tests are in syscalls/ directory.

> 3, Where to check those out put standards, taking yocto package test as a
> small comprehension, that standard was ( PASS|FAIL|SKIP): Test info. How is
> it like in LTP?

Generally tests in LTP can have more than one result so you can't really
map a test result to a single value. It's common for a test to report
several TPASS and one or two TCONF for example. That means that most of
the test was executed but small part was skipped.

The TINFO messages are additional information that may be useful when
test fail, nothing more.

The overall result is reported in the test exit value as bitflags, which
is what is used by the runltp script to produce the result files.

And by the way, we are working on replacement for the runltp script
which is called runltp-ng, maybe you should have a look at that before
you try to write your own solution.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] Question about log format.

[Richard Palethorpe]

Hello,

Xinkuan Yan <lucky33newman@gmail.com> writes:

> Hello list,
>
> I'm trying to do a concise parsing of the LTP report.
>
> I don't have much experience and I'm not familiar with LTP, so I'm going to
> take #math as an example, for it is not complex.
>
> Run test command:
>
> ./ltp –f math –q
>
> I have three questions now,  taking the one-line output result below as an
> example:
>
> float_iperb 1 TPASS: Test passed

This is a test using the old API, so not the best example.

>
> 1, It is clear that TPASS means it passed, but what is the number 1
> standing for.

I don't know to be honest. It was removed in the new test API.

>
> 2, Why this line showed up twice? ( PS: Between these two lines are many
> TINFO.)

Possibly because there are multiple iterations of the test. Possibly
some paramters are changed between iterations.

Only way to find out is to look for tst_resm calls in the source code.

>
> 3, Where to check those out put standards, taking yocto package test as a
> small comprehension, that standard was ( PASS|FAIL|SKIP): Test info. How is
> it like in LTP?

https://github.com/linux-test-project/ltp/wiki/C-Test-API#12-basic-test-interface

Note taht the new API output looks more like the following:

sh-5.2# ./bpf_prog07
tst_buffers.c:55: TINFO: Test is using guarded buffers
tst_taint.c:116: TCONF: Ignoring already set kernel warning taint
tst_test.c:1560: TINFO: Timeout per run is 0h 00m 30s
bpf_common.c:17: TINFO: Raising RLIMIT_MEMLOCK to 10485760
tst_capability.c:29: TINFO: Dropping CAP_SYS_ADMIN(21)
tst_capability.c:29: TINFO: Dropping CAP_BPF(39)
bpf_common.c:39: TCONF: Hint: check also /proc/sys/kernel/unprivileged_bpf_disabled
bpf_common.c:41: TCONF: bpf() requires CAP_SYS_ADMIN or CAP_BPF on this system: EPERM (1)

Summary:
passed   0
failed   0
broken   0
skipped  3
warnings 0

sh-5.2# ./bpf_prog01
tst_buffers.c:55: TINFO: Test is using guarded buffers
tst_test.c:1560: TINFO: Timeout per run is 0h 00m 30s
bpf_common.c:17: TINFO: Raising RLIMIT_MEMLOCK to 10485760
bpf_common.c:114: TPASS: Loaded program
bpf_prog01.c:95: TPASS: val = 1
...

So the format is like:

<file_name>:<line>: <ttype>: <message>

-- 
Thank you,
Richard.

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] Question about log format.

[Xinkuan Yan]

Hi,

Sorry for the late reply.

Your answers helped me so much from getting lost in ltp, wooops, this
massive Frankenstain made me go mad when I design the core parser function.

Wish runltp-ng version a great job! Looking forward to see that.

Cheers,
Yan

Richard Palethorpe <rpalethorpe@suse.de> 于2022年12月12日周一 22:45写道：

> Hello,
>
> Xinkuan Yan <lucky33newman@gmail.com> writes:
>
> > Hello list,
> >
> > I'm trying to do a concise parsing of the LTP report.
> >
> > I don't have much experience and I'm not familiar with LTP, so I'm going
> to
> > take #math as an example, for it is not complex.
> >
> > Run test command:
> >
> > ./ltp –f math –q
> >
> > I have three questions now,  taking the one-line output result below as
> an
> > example:
> >
> > float_iperb 1 TPASS: Test passed
>
> This is a test using the old API, so not the best example.
>
> >
> > 1, It is clear that TPASS means it passed, but what is the number 1
> > standing for.
>
> I don't know to be honest. It was removed in the new test API.
>
> >
> > 2, Why this line showed up twice? ( PS: Between these two lines are many
> > TINFO.)
>
> Possibly because there are multiple iterations of the test. Possibly
> some paramters are changed between iterations.
>
> Only way to find out is to look for tst_resm calls in the source code.
>
> >
> > 3, Where to check those out put standards, taking yocto package test as a
> > small comprehension, that standard was ( PASS|FAIL|SKIP): Test info. How
> is
> > it like in LTP?
>
>
> https://github.com/linux-test-project/ltp/wiki/C-Test-API#12-basic-test-interface
>
> Note taht the new API output looks more like the following:
>
> sh-5.2# ./bpf_prog07
> tst_buffers.c:55: TINFO: Test is using guarded buffers
> tst_taint.c:116: TCONF: Ignoring already set kernel warning taint
> tst_test.c:1560: TINFO: Timeout per run is 0h 00m 30s
> bpf_common.c:17: TINFO: Raising RLIMIT_MEMLOCK to 10485760
> tst_capability.c:29: TINFO: Dropping CAP_SYS_ADMIN(21)
> tst_capability.c:29: TINFO: Dropping CAP_BPF(39)
> bpf_common.c:39: TCONF: Hint: check also
> /proc/sys/kernel/unprivileged_bpf_disabled
> bpf_common.c:41: TCONF: bpf() requires CAP_SYS_ADMIN or CAP_BPF on this
> system: EPERM (1)
>
> Summary:
> passed   0
> failed   0
> broken   0
> skipped  3
> warnings 0
>
> sh-5.2# ./bpf_prog01
> tst_buffers.c:55: TINFO: Test is using guarded buffers
> tst_test.c:1560: TINFO: Timeout per run is 0h 00m 30s
> bpf_common.c:17: TINFO: Raising RLIMIT_MEMLOCK to 10485760
> bpf_common.c:114: TPASS: Loaded program
> bpf_prog01.c:95: TPASS: val = 1
> ...
>
> So the format is like:
>
> <file_name>:<line>: <ttype>: <message>
>
> --
> Thank you,
> Richard.
>

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp