Re: [PATCH 6/6] u-boot: Rework signing to remove interdependencies

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Wed, 2022-10-26 at 12:49 -0400, Sean Anderson wrote:
> On 10/26/22 10:08, Richard Purdie wrote:
> > This looks like a pretty nice improvement, I've been worried about how
> > entangled all this was becoming. Is there anything better we could do
> > with testing of this code? We rely increasingly on the tests to spot
> > regressions and I doubt the testcases we have cover the code
> > complexity.
> 
> FWIW, I found the test suite to be fairly helpful when debugging this
> whole process, if agonizingly slow. I found many bugs in my initial
> implementation which were uncovered by the test suites,
> 
> The best way to extend the test suite would probably be to have QEMU try
> to boot using the Linux and U-Boot. This would help cases where all the
> artifacts are created correctly but e.g. the wrong U-Boot is packaged.
> However, I am rather loathe to add more test cases like this, because it
> already takes around 10 minutes (with sstate!) to run the tests. This
> basically kills the speed of iterative development.

Out of interest how were you running the tests? Did you look at the
parallelism options? Were you just running specific tests or specific
suites of tests?

I do think we need something like you describe and adding a linux+uboot
approach is something I've wanted to do for a long time. Usually the
time is taken on our automated CI rather than impacting users, unless
they're working in that area at which point the tests are hopefully
helpful.

We haven't really had anyone try and optimise the tests either, I'm
sure there will be things in there which can help. Please don't let the
speed put you off trying to improve things and extend our coverage!

Cheers,

Richard