Re: [OE-core] [PATCH] patchtest: Add further information for failed testcases

[Trevor Gamblin <tgamblin@baylibre.com>]

On 2024-02-16 11:19, Simone Weiß wrote:
> On Thu, 2024-02-15 at 22:10 +0000, Richard Purdie wrote:
>> On Thu, 2024-02-15 at 21:39 +0000, Simone Weiß wrote:
>>> From: Simone Weiß <simone.p.weiss@posteo.com>
>>>
>>> Add more information to log messages when a test case fails.
>>> Still keep it short and mostly reference the documentation. Reasson is
>>> that
>>> documentation should already contain the needed information, do not
>>> duplicate
>>> it here, so we also do not need to update here should the doc/policy
>>> change.
>>>
>>> Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com>
>> This does get a little tricky since the responses from patchtest on our
>> test infrastructure has a character limit for security reasons.
>>
>> I have wondered if we do need to change it to some kind of code
>> response which can then be translated back into real text.
>>
> What could be easily done is also to remove some kind of information that
> is kinda duplicated from the logs: e.g.
>
> FAIL: test commit message presence: Please include a commit message on
> your patch explaining the change
> (test_mbox.TestMbox.test_commit_message_presence)
>
> basically duplicates test_commit_message_presence (with and w/o _), that
> is not really needed IMO.
>
> Also the reference links could be given instead - if any testcases failed
> - in  a bundled fashion by adopting the suggestions in patchtest-send-
> result and for local runs in patchtest's run function

We could reference the wiki page: 
https://wiki.yoctoproject.org/wiki/Patchtest

We'll need to update it too, though. I put it together to have 
*something* when patchtest was brought back online, but there's 
definitely room for improvement.

I think a combination of both suggestions, i.e. a much shorter "FAIL: 
test commit message presence (code: 1234)" sort of approach along with a 
link to the wiki page is the best here.

What are your thoughts?

>
> Cheers,
> Simone
>> The reason for the character limit is to stop abuse of the VMs that run
>> the tests, you can't mine bitcoin and pass them out as the failure
>> message for example.
>>
>> Cheers,
>>
>> Richard
>>
>>
>>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#195780): https://lists.openembedded.org/g/openembedded-core/message/195780
> Mute This Topic: https://lists.openembedded.org/mt/104382125/7611679
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [tgamblin@baylibre.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>