Using kernelCI infrastructure for firmware testing

Using kernelCI infrastructure for firmware testing

[patrick.rudolph]

Hi folks,
this is an attempt to improve firmware testing by using the
infrastructure and knowledge of the kernelci community. If you think
this is not the right place, please point me in the right direction.

I'm a coreboot[1] developer trying to make sure that the master
branch[2] doesn't regress. Currently there's no public firmware
testing, only internal validation suites used by some companies that
lack direct and automated feedback before a commit is actually merged.
As this isn't a coreboot only topic, but applies to all open source
"bios vendors", I added the u-boot project in CC as well.

For me firmware testing looks pretty similar to kernel testing:
* flash firmware to test
* boot a known good linux kernel
* run tests in userspace and verify hardware/software works as expected

On the hardware side we have boards in our lab that allow remote power
cycling and firmware flashing. It is attached to self hosted stock
LAVA2018. But as we are firmware engineers, we don't want to deal with
the administration of servers.

Here are a few questions for you:
* Would it make sense to also cover open source firmware tests on kernelci?
* Do you build the linux images yourself?
* Would you accept firmware images generated by a third party?
* Can anybody get an account for the LAVA server to run firmware test?
* What communication channels do you recommend?
* Will there be meetings or conferences to get in contact with the
community to talk about this?

[1]: https://coreboot.org/
[2]: https://review.coreboot.org/

Kind Regards,
Patrick Rudolph

9elements GmbH, Kortumstraße 19-21, 44787 Bochum, Germany
Email:  patrick.rudolph@9elements.com
Phone:  +49 234 / 68 94 188

Sitz der Gesellschaft: Bochum
Handelsregister: Amtsgericht Bochum, HRB 13207
Geschäftsführung: Eray Basar, Sebastian Deutsch

Re: [U-Boot] Using kernelCI infrastructure for firmware testing

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 2035 bytes --]

On Fri, Nov 08, 2019 at 10:35:10AM +0100, Patrick Rudolph wrote:

> Hi folks,
> this is an attempt to improve firmware testing by using the
> infrastructure and knowledge of the kernelci community. If you think
> this is not the right place, please point me in the right direction.
> 
> I'm a coreboot[1] developer trying to make sure that the master
> branch[2] doesn't regress. Currently there's no public firmware
> testing, only internal validation suites used by some companies that
> lack direct and automated feedback before a commit is actually merged.
> As this isn't a coreboot only topic, but applies to all open source
> "bios vendors", I added the u-boot project in CC as well.
> 
> For me firmware testing looks pretty similar to kernel testing:
> * flash firmware to test
> * boot a known good linux kernel
> * run tests in userspace and verify hardware/software works as expected
> 
> On the hardware side we have boards in our lab that allow remote power
> cycling and firmware flashing. It is attached to self hosted stock
> LAVA2018. But as we are firmware engineers, we don't want to deal with
> the administration of servers.
> 
> Here are a few questions for you:
> * Would it make sense to also cover open source firmware tests on kernelci?
> * Do you build the linux images yourself?
> * Would you accept firmware images generated by a third party?
> * Can anybody get an account for the LAVA server to run firmware test?
> * What communication channels do you recommend?
> * Will there be meetings or conferences to get in contact with the
> community to talk about this?
> 
> [1]: https://coreboot.org/
> [2]: https://review.coreboot.org/

Over in U-Boot, yes, we have some test suites that we run on real HW as
well as QEMU-based hardware.  And historically I've talked with Kevin
once or twice about testing a current U-Boot on various easy-to-recover
boards rather than just what it shipped with, but things never moved
past the talking about it stage.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Using kernelCI infrastructure for firmware testing

[Kevin Hilman]

"Patrick Rudolph" <patrick.rudolph@9elements.com> writes:

> Hi folks,
> this is an attempt to improve firmware testing by using the
> infrastructure and knowledge of the kernelci community. If you think
> this is not the right place, please point me in the right direction.
>
> I'm a coreboot[1] developer trying to make sure that the master
> branch[2] doesn't regress. Currently there's no public firmware
> testing, only internal validation suites used by some companies that
> lack direct and automated feedback before a commit is actually merged.
> As this isn't a coreboot only topic, but applies to all open source
> "bios vendors", I added the u-boot project in CC as well.
>
> For me firmware testing looks pretty similar to kernel testing:
> * flash firmware to test
> * boot a known good linux kernel
> * run tests in userspace and verify hardware/software works as expected
>
> On the hardware side we have boards in our lab that allow remote power
> cycling and firmware flashing. It is attached to self hosted stock
> LAVA2018. But as we are firmware engineers, we don't want to deal with
> the administration of servers.
>
> Here are a few questions for you:
> * Would it make sense to also cover open source firmware tests on kernelci?

Yes.  We've talked about this a few times over the years, and all the
infra and tooling we have built for kernelCI would be well suited for
firmware testing.

The catch is mostly in time/resources invested.  For now, we are
primarily focused on kernel testing, but if there are contributors that
want to extend this to firmware, we are an open-source project and
welcome the contributions.

I would say the primary challenge here is that each SoC family, and
sometimes even each board has unique challenges to (re)flashing the
firmware as well as the ability to recover from non-working firmware.

This presents a problem not necessariuly for KernelCI but for the lab
admins for that hardware, and especially those writing the LAVA
device-types for those platforms.

I've added the main LAVA developer Remi Durrafort to Cc since he's been
doing a lot of work in LAVA to make firmware updates and recovery mode
usable from LAVA.  Recent versions of LAVA seem to be growing better
support for this:
https://docs.lavasoftware.org/lava/bootloaders.html

> * Do you build the linux images yourself?

We build all the linux kernel images, but rely on the lab admins to have
build/flashed working firmware and bootloaders.

> * Would you accept firmware images generated by a third party?

As long as the source is available and can be (re)built, that's
typically fine.

> * Can anybody get an account for the LAVA server to run firmware test?

That's up to each LAVA lab administrator.  Today, KernelCI uses a
distributed set of (mostly LAVA) labs.  Each lab admin has given
priveleges to KernelCI to send jobs.

> * What communication channels do you recommend?

This mailing list is a good place to start, but to be completely
honest, firmware testing is relatively low on our current list of
priorites.  But as I said above, it's feature we will welcome with open
arms, even if it's not a feature that the current set of developers will
be very active on.

I would say your first step is probably to get some support for the
devices you care about into upstream LAVA.  Once that is working, we
would work on how KernelCI could start generating jobs for those devices.

> * Will there be meetings or conferences to get in contact with the
> community to talk about this?

We just had an automated-testing summit connected to Embedded Linux
Conference Europe last month, where many of us were gathered.

Remi even gave a talk on bootloader testing with LAVA, maybe he can
share the slides for that.

Kevin