Illustration of good and bad QA test output

[Richard Purdie <richard.purdie@linuxfoundation.org>]

A while back, I asked for a review of test output when things fail as it
was causing problems. I suspect some people don't really understand why
this is a big deal. I'd therefore like to illustrate this with a new
example:

https://autobuilder.yoctoproject.org/main/builders/nightly-oe-selftest/builds/180/steps/Running%20oe-selftest/logs/stdio

This is a failure we encountered on the autobuilder. It basically tells
me that "1 is not 0".  I did see this failure in master-next but I had
no what to know what was causing it, whether it was a patch in -next or
whether it came from somewhere else. I therefore had to move things
forward and merged -next.

Locally, I added this change:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=c8134250f72d5102f96137265ae71cf8f2eff5a9

In amongst the build output this shows when the test fails, I see:

WARNING: Failed to fetch URL file://d1/sstate:m4::1.4.17:r0::3:d1c56d6fa574f1093f4be30cb90ac127_populate_lic.tgz.sig, attempting MIRRORS if available
ERROR: Fetcher failure: Unable to find file file://d1/sstate:m4::1.4.17:r0::3:d1c56d6fa574f1093f4be30cb90ac127_populate_lic.tgz.sig anywhere. The paths that were searched were:
    /media/build1/poky/build/temp_sstate_20150907190612
    /media/build1/poky/build/temp_sstate_20150907190612
NOTE: recipe m4-1.4.17-r0: task do_populate_lic_setscene: Succeeded

and from that error and my knowledge of what was in -next, I can be
pretty sure this comes from:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=e3feac122b6baa67a6e75a99da6e8834f0f2a7b0

and I now know who to blame (sorry Ross!). The issue is that this is now
in master and we have the error in a much more serious place. If the QA
code was showing better information in the case of failure, this would
never have made it into master.

I mention this since I'm hoping a practical example of how the test
failures influence decisions and make a difference to the project might
encourage people to pay more attention to these details.

Cheers,

Richard