[LTP] Issue with the reporting of failed tests

[Mason <slash.tmp@free.fr>]

On 25/04/2016 13:40, Mason wrote:
> On 25/04/2016 10:04, Mason wrote:
>> On 22/04/2016 15:29, Mason wrote:
>>> On 21/04/2016 19:02, Mason wrote:
>>>> On 21/04/2016 17:42, Mason wrote:
>>>>> On 21/04/2016 17:30, Cyril Hrubis wrote:
>>>>>> Hi!
>>>>>>> Calling fflush does appear to solve the problem:
>>>>>>>
>>>>>>> # grep -c FAIL result-log.5160
>>>>>>> 38
>>>>>>>
>>>>>>> # wc -l result-failed.5160
>>>>>>> 38 result-failed.5160
>>>>>>>
>>>>>>> Does this mean no one ever reads FAILCMDFILE?!
>>>>>>
>>>>>> Just to have complete image here. I've discussed this a bit more at #ltp
>>>>>> irc channel and the reason why we do not see this bug often is that the
>>>>>> exec() that is done by the child after pan forks must fail (for example
>>>>>> since bash is missing) then the code proceeds to exit(errno) line which
>>>>>> flushes the buffers on open files...
>>>>>
>>>>> Note: in my latest LTP run, I disabled memory overcommit, and pan
>>>>> failed to launch several tests:
>>>>>
>>>>> pan(5163): fork failed (tag proc01).  errno:12  Cannot allocate memory
>>>>> pan(5163): fork failed (tag fs_racer).  errno:12  Cannot allocate memory
>>>>> pan(5163): fork failed (tag quota_remount_test01).  errno:12  Cannot allocate memory
>>>>> pan(5163): fork failed (tag isofs).  errno:12  Cannot allocate memory
>>>>> pan(5163): fork failed (tag fs_perms01).  errno:12  Cannot allocate memory
>>>>> [747 similar lines]
>>>>>
>>>>> /tmp is a ramfs. Maybe something fills it up...
>>>>
>>>> With overcommit enabled, all tests complete, but some problems remain:
>>>>
>>>> $ grep -c FAIL result-log.24583 
>>>> 85
>>>>
>>>> $ wc -l result-failed.24583
>>>> 157 result-failed.24583
>>>>
>>>> $ sort result-failed.24583 | uniq | wc -l
>>>> 83
>>>>
>>>> I'm confused as to what else is causing this...
>>>
>>> I am now testing yet another LTP run with the following patch applied.
>>
>> There still is an inconsistency between result-log and result-failed.
>>
>> $ grep -c FAIL result-log.30032
>> 119
>>
>> $ wc -l result-failed.30032
>> 319 result-failed.30032
>>
>> $ sort result-failed.30032 | uniq | wc -l
>> 117
>>
>> Either I didn't use the modified ltp-pan binary, or the change has no effect.
>> (I will print something to stderr to make sure.)
> 
> I do see an improvement now: no more duplicates. However, it seems
> 2 tests are missing from result-failed: aio01, aio02.

These two tests are not being built.
FILTER_OUT_DIRS		:= $(if $(AIO_LIBS),,aio ltp-aiodio)

so pan fails to launch them, obviously.
pan(1130): execvp of 'aio01' (tag aio01) failed.  errno:2  No such file or directory
pan(1130): execvp of 'aio_tio' (tag aio02) failed.  errno:2  No such file or directory

I don't think these should be considered failures?

Regards.