Re: [OE-core] [PATCH] patchtest: Add further information for failed testcases

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Fri, 2024-02-16 at 11:43 -0500, Trevor Gamblin wrote:
> 
> On 2024-02-16 11:19, Simone Weiß wrote:
> > On Thu, 2024-02-15 at 22:10 +0000, Richard Purdie wrote:
> > > On Thu, 2024-02-15 at 21:39 +0000, Simone Weiß wrote:
> > > > From: Simone Weiß <simone.p.weiss@posteo.com>
> > > > 
> > > > Add more information to log messages when a test case fails.
> > > > Still keep it short and mostly reference the documentation.
> > > > Reasson is
> > > > that
> > > > documentation should already contain the needed information, do
> > > > not
> > > > duplicate
> > > > it here, so we also do not need to update here should the
> > > > doc/policy
> > > > change.
> > > > 
> > > > Signed-off-by: Simone Weiß <simone.p.weiss@posteo.com>
> > > This does get a little tricky since the responses from patchtest
> > > on our
> > > test infrastructure has a character limit for security reasons.
> > > 
> > > I have wondered if we do need to change it to some kind of code
> > > response which can then be translated back into real text.
> > > 
> > What could be easily done is also to remove some kind of
> > information that
> > is kinda duplicated from the logs: e.g.
> > 
> > FAIL: test commit message presence: Please include a commit message
> > on
> > your patch explaining the change
> > (test_mbox.TestMbox.test_commit_message_presence)
> > 
> > basically duplicates test_commit_message_presence (with and w/o _),
> > that
> > is not really needed IMO.
> > 
> > Also the reference links could be given instead - if any testcases
> > failed
> > - in  a bundled fashion by adopting the suggestions in patchtest-
> > send-
> > result and for local runs in patchtest's run function
> 
> We could reference the wiki page: 
> https://wiki.yoctoproject.org/wiki/Patchtest
> 
> We'll need to update it too, though. I put it together to have 
> *something* when patchtest was brought back online, but there's 
> definitely room for improvement.
> 
> I think a combination of both suggestions, i.e. a much shorter "FAIL:
> test commit message presence (code: 1234)" sort of approach along
> with a 
> link to the wiki page is the best here.
> 
> What are your thoughts?

Something like that should be ok, I would have it fill out what code:
1234 means though so people don't have to go to the wiki every time.

Cheers,

Richard