Re: [Fuego] Fuego's version up and other changes

[Kevin Hilman <khilman@baylibre.com>]

"Bird, Timothy" <Tim.Bird@sony.com> writes:

>> -----Original Message-----
>> From: Kevin Hilman on Thursday, February 02, 2017 12:52 PM
>> "Bird, Timothy" <Tim.Bird@sony.com> writes:
>> 
>> [...]
>> 
>> > I haven't seen your lava integration slides yet, but it sounds like
>> > they have some elaborate setup and teardown operations as well.
>> 
>> Speaking of LAVA (and hijacking the thread a bit, since I just
>> subscribed...)
>> 
>> I'm new to fuego and would love to see/hear more about the LAVA
>> integration status/plans/future etc. (Unfortunately, I won't be a FOSDEM
>> to hear Jan-Simon's talk.)
>
> He'll be giving the talk at ELC, so if you're coming to that you can
> see it there (or a few weeks later on video).
>
>> Fuego seems OK in terms of running tests, but actually adding boards is,
>> well... less than what I was hoping for.
>>
>> There are many labs out there with lots of different boards hooked up to
>> LAVA already, so having a way for Fuego to submit tests and extract
>> results from LAVA seems like a good area for future work.
>
> Ok - now seems like as good a time as any to start rambling on about the
> Fuego roadmap and priorities... :-)
>
> Yeah - I've consciously NOT put a lot of work into Fuego's board handling,
> since there are other systems (including LAVA) that do this already.
> Right now, Fuego supports software reboot, but doesn't have an API
> for the kind of hardware control of power, reboot, reflashing, serial console,
> etc. needed for kernel bring-up testing, like what KernelCI does.
>
> Since KernelCI is doing a good job in that space (Continuous Integration
> kernel boot testing at top of tree on a matrix of boards), I've been trying
> to position Fuego as more of a distribution tester.  So, for now we're assuming
> that the kernel boot and networking are stable, and Fuego is focused on
> testing features and packages.

Does Fuego handle the distro install/upgrade piece?  I couldn't tell so
far if it does.

Anyways, I agree that getting a broad suite of tests on a stable
kernel/rootfs is the right focus for testing, at least for starters.

While kernel CI does lots of upstream trees, it also does stable trees
(and the stable-rc tree) where things should, by definition, be
mostly stable, so we've been aiming at stable for broad testing, but
also care about a rather wide variety of hardware, across several arch
(currently x86, arm, arm64 and MIPS.)

Also, focusing testing on stable means better starting point for distro
kernels too.  In addition, being able to compare a test report from a
distro kernel to the stable kernel it was based on would be very useful
data.

> Right now, we're pretty reliant on a stable ssh, shell and filesystem.  We've
> lately been workingon improving our Jenkins integration (decoupling from a
> specific version of Jenkins, or even from Jenkins itself, with our own CLI
> test tool), and migrating towards a standalone test package format
> (so all tests don't have to be in our repository, and we can distribute
> tests independently from the Fuego core system).

In scaling up kernel CI over the last year or so, de-coupling from
Jenkins is a very good idea. ;)

> Big areas to follow are:
>  - canonical test output format
>  - ability to gather test results from nodes
>  - ability to download test packages from a "store"

Thanks to a pointer from Jan-Simon, I started looking at Avocado
recently.  In terms of a standalone CLI tool, it seems to do all of this
(except maybe the store part.) 

Avocado also seems ot have a great handle on testing where there's more
than one "entity" involved (e.g. KVM boot with a host and multiple
guests.)  Not sure if Fuego has that yet?

Curious what your opinions of Avocado are and how it might fit into the
Fuego vision?  I've only briefly started looking at both Fuego and
Avocado, so I don't really have a good grip yet on where there may be
overlap.

> The area I'd like Fuego to focus on, that I don't see other Linux test
> frameworks doing, is actually building out a complement of tests.
> I'd like Fuego to be a "test distribution" with lots of tests for different
> aspects of the system (file system, networking, realtime, power management,
> etc.)  I see it as important to get the package system and test API
> right, before doing a concerted push to create all the different tests 
> in different areas. This is why the current focus is where it is.

I fully agree.  Writing and running tests is easy.  The hard part is
getting them packaged in a way that's usuable across a wide range of
hardware, and parsing/processing the results into useful reports that
include historical/regression analysis.  Also making the tests tolerant
of interfaces/modules etc. that may or may not be present.

> Our first priority is as a distribution tester, where someone (a Linux
> Foundation project, a semiconductor vendor, or an internal team
> at a big company) has produced a distribution of Linux, and wants to
> validate that it has few bugs, and continues working as intended when
> delivered to other groups.  The ideal is that the whole test framework
> can be delivered with the distribution, and the distro customer can
> continue to validate for themselves that nothing has broken in the distro
> while thy develop their end product (and modify parts of the distro).

That sounds like a good priority.

What distro(s) are you focused on now?  It seems like there should be
some definition of minimal/baseline distro (e.g. yocto
core-image-minimal) for starters.

> Anyway - I'm also interested in Jan-Simon's LAVA integration.
> It would be great to be able to run Fuego tests in current KernelCI
> labs.

Exactly.  If that was a possibility, we could offer a broad range of 
hardware for experimentation with Fuego very easily.

Kevin