Re: [OE-core] [PATCH] patchtest: Add further information for failed testcases

["Simone Weiß" <simone.weiss@posteo.net>]

On Fri, 2024-02-16 at 11:43 -0500, Trevor Gamblin wrote:
> 
> On 2024-02-16 11:19, Simone Weiß wrote:
> > On Thu, 2024-02-15 at 22:10 +0000, Richard Purdie wrote:
> > > On Thu, 2024-02-15 at 21:39 +0000, Simone Weiß wrote:
> > > > From: Simone Weiß <simone.p.weiss@posteo.com>
> > > > 
> > > > Add more information to log messages when a test case fails.
> > > > Still keep it short and mostly reference the documentation.
> > > > Reasson is
> > > > that
> > > > documentation should already contain the needed information, do
> > > > not
> > > > duplicate
> > > > it here, so we also do not need to update here should the
> > > > doc/policy
> > > > change.
> > > > 
> > > > Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com>
> > > This does get a little tricky since the responses from patchtest on
> > > our
> > > test infrastructure has a character limit for security reasons.
> > > 
> > > I have wondered if we do need to change it to some kind of code
> > > response which can then be translated back into real text.
> > > 
> > What could be easily done is also to remove some kind of information
> > that
> > is kinda duplicated from the logs: e.g.
> > 
> > FAIL: test commit message presence: Please include a commit message on
> > your patch explaining the change
> > (test_mbox.TestMbox.test_commit_message_presence)
> > 
> > basically duplicates test_commit_message_presence (with and w/o _),
> > that
> > is not really needed IMO.
> > 
> > Also the reference links could be given instead - if any testcases
> > failed
> > - in  a bundled fashion by adopting the suggestions in patchtest-send-
> > result and for local runs in patchtest's run function
> 
> We could reference the wiki page: 
> https://wiki.yoctoproject.org/wiki/Patchtest
> 
> We'll need to update it too, though. I put it together to have 
> *something* when patchtest was brought back online, but there's 
> definitely room for improvement.
> 
> I think a combination of both suggestions, i.e. a much shorter "FAIL: 
> test commit message presence (code: 1234)" sort of approach along with a
> link to the wiki page is the best here.
> 
> What are your thoughts?
I would also combine all proposals:
- Update the wikipage, but where possible cross reference the
documentation to avoid duplications. This we can do anyhow.
- shorten the general test logs (remove duplications of test name)
- Add the link to the wikipage in patchtest-send-result or for local runs
at the end of a test run.

Do we really need the code 1234 approach if we just log 

FAIL: testcase_name

and the map the names in the wikipage to the needed info?

> > 
> > Cheers,
> > Simone
> > > The reason for the character limit is to stop abuse of the VMs that
> > > run
> > > the tests, you can't mine bitcoin and pass them out as the failure
> > > message for example.
> > > 
> > > Cheers,
> > > 
> > > Richard
> > > 
> > > 
> > > 
> > 
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#195780):
> > https://lists.openembedded.org/g/openembedded-core/message/195780
> > Mute This Topic: https://lists.openembedded.org/mt/104382125/7611679
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe:
> > https://lists.openembedded.org/g/openembedded-core/unsub [
> > tgamblin@baylibre.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >