KernelCI modular pipeline design document

KernelCI modular pipeline design document

[Guillaume Tucker]

Based on previous discussions around the topic of expanding
KernelCI with alternatives to Jenkins, LAVA, our Mongo DB backend
and current frontend, I've made this first version of a document
with design decisions as to how we can make it happen:

  https://docs.google.com/document/d/15F42HdHTO6NbSL53_iLl77lfe1XQKdWaHAf7XCNkKD8/edit?usp=sharing

It doesn't cover all the details of how to actually implement
things but sets some general lines which should be enough to keep
us going towards a common objective.  It would be great if we
could adjust it whenever necessary and agree to use it as a
reference.

Please let me know what you think, and if you need to review or
edit it so I'll give you permissions.

Best wishes,
Guillaume

Re: KernelCI modular pipeline design document

["Jan-Simon Möller"]

[-- Attachment #1: Type: text/html, Size: 2002 bytes --]

Re: KernelCI modular pipeline design document

[Milosz Wasilewski]

On Fri, 8 Feb 2019 at 16:33, "Jan-Simon Möller" <JanSimon.Moeller@gmx.de> wrote:
>
> Please do not just think of 'building the kernel' - ppl also want to test and record/visualize userspace tests (aka custom-built kernel+filesystem).
> Yes, the linux kernel is the main focus for kernelci.org . But let's keep the other possible use-cases in mind.
>

I don't think this is a good idea. Having other use cases in mind
takes you down a very deep rabbit hole. Creating a general purpose
result reporting tool is hard. Kernelci reports on kernel results and
(as I understand) does it right. You can use these tools for other
purposes but that's a different story.

milosz

Re: KernelCI modular pipeline design document

[Kevin Hilman]

"Milosz Wasilewski" <milosz.wasilewski@linaro.org> writes:

> On Fri, 8 Feb 2019 at 16:33, "Jan-Simon Möller" <JanSimon.Moeller@gmx.de> wrote:
>>
>> Please do not just think of 'building the kernel' - ppl also want to test and record/visualize userspace tests (aka custom-built kernel+filesystem).
>> Yes, the linux kernel is the main focus for kernelci.org . But let's keep the other possible use-cases in mind.
>>
>
> I don't think this is a good idea. Having other use cases in mind
> takes you down a very deep rabbit hole. Creating a general purpose
> result reporting tool is hard. Kernelci reports on kernel results and
> (as I understand) does it right. You can use these tools for other
> purposes but that's a different story.

Yes, we're focused on kernel-focused testing, but there are multiple
ways to test the kernel, most of which require a combination of kernel +
userspace/distro.

I think Jan-Simon's point is that as we evolve, while we might focus on
buildroot/debian, we chould not design things in a way that prohibit
using the kernelCI code (and infra) for other types of kernel-focused
testing (e.g. Yocto, other distros, etc.)

Stated differently, the current *service* provided by kernelCI.org is
kernel-focused testing.  But, we could (and should, IMO) write the
*software* behind that service in a way that it could be used more
generically if desired.

Kevin

Re: KernelCI modular pipeline design document

[Milosz Wasilewski]

On Mon, 11 Feb 2019 at 18:27, Kevin Hilman <khilman@baylibre.com> wrote:
>
> "Milosz Wasilewski" <milosz.wasilewski@linaro.org> writes:
>
> > On Fri, 8 Feb 2019 at 16:33, "Jan-Simon Möller" <JanSimon.Moeller@gmx.de> wrote:
> >>
> >> Please do not just think of 'building the kernel' - ppl also want to test and record/visualize userspace tests (aka custom-built kernel+filesystem).
> >> Yes, the linux kernel is the main focus for kernelci.org . But let's keep the other possible use-cases in mind.
> >>
> >
> > I don't think this is a good idea. Having other use cases in mind
> > takes you down a very deep rabbit hole. Creating a general purpose
> > result reporting tool is hard. Kernelci reports on kernel results and
> > (as I understand) does it right. You can use these tools for other
> > purposes but that's a different story.
>
> Yes, we're focused on kernel-focused testing, but there are multiple
> ways to test the kernel, most of which require a combination of kernel +
> userspace/distro.
>
> I think Jan-Simon's point is that as we evolve, while we might focus on
> buildroot/debian, we chould not design things in a way that prohibit
> using the kernelCI code (and infra) for other types of kernel-focused
> testing (e.g. Yocto, other distros, etc.)

I guess 'kernel focused' is the key here. I read 'other possible use
cases' as a request to build a general purpose reporting tool.

>
> Stated differently, the current *service* provided by kernelCI.org is
> kernel-focused testing.  But, we could (and should, IMO) write the
> *software* behind that service in a way that it could be used more
> generically if desired.

I've been trying to do that for last couple of years with no major
success. It might be just me or the fact that it's really hard. Once
you try to start reporting on android runtime or openstack things get
really complicated.

So if you consider running tests that exercise different parts of
kernel (like graphics, scheduler...) as 'other use cases' than yes,
KCI software should be able to do that. But I don't think going the
route of 'general purpose reporting tool' is the right decision.

milosz

Re: KernelCI modular pipeline design document

[Kevin Hilman]

Milosz Wasilewski <milosz.wasilewski@linaro.org> writes:

> On Mon, 11 Feb 2019 at 18:27, Kevin Hilman <khilman@baylibre.com> wrote:
>>
>> "Milosz Wasilewski" <milosz.wasilewski@linaro.org> writes:
>>
>> > On Fri, 8 Feb 2019 at 16:33, "Jan-Simon Möller" <JanSimon.Moeller@gmx.de> wrote:
>> >>
>> >> Please do not just think of 'building the kernel' - ppl also want to test and record/visualize userspace tests (aka custom-built kernel+filesystem).
>> >> Yes, the linux kernel is the main focus for kernelci.org . But let's keep the other possible use-cases in mind.
>> >>
>> >
>> > I don't think this is a good idea. Having other use cases in mind
>> > takes you down a very deep rabbit hole. Creating a general purpose
>> > result reporting tool is hard. Kernelci reports on kernel results and
>> > (as I understand) does it right. You can use these tools for other
>> > purposes but that's a different story.
>>
>> Yes, we're focused on kernel-focused testing, but there are multiple
>> ways to test the kernel, most of which require a combination of kernel +
>> userspace/distro.
>>
>> I think Jan-Simon's point is that as we evolve, while we might focus on
>> buildroot/debian, we chould not design things in a way that prohibit
>> using the kernelCI code (and infra) for other types of kernel-focused
>> testing (e.g. Yocto, other distros, etc.)
>
> I guess 'kernel focused' is the key here. I read 'other possible use
> cases' as a request to build a general purpose reporting tool.
>
>>
>> Stated differently, the current *service* provided by kernelCI.org is
>> kernel-focused testing.  But, we could (and should, IMO) write the
>> *software* behind that service in a way that it could be used more
>> generically if desired.
>
> I've been trying to do that for last couple of years with no major
> success. It might be just me or the fact that it's really hard. Once
> you try to start reporting on android runtime or openstack things get
> really complicated.
>
> So if you consider running tests that exercise different parts of
> kernel (like graphics, scheduler...) as 'other use cases' than yes,
> KCI software should be able to do that. But I don't think going the
> route of 'general purpose reporting tool' is the right decision.

Agreed.  We should stay focused on *kernel* CI.

But, we don't want to (artificially) limit the types of
tools/distros/stacks we use to beat up on the kernel.

Kevin

Re: KernelCI modular pipeline design document

[Guillaume Tucker]

[-- Attachment #1: Type: text/plain, Size: 3970 bytes --]

On Tue, Feb 12, 2019 at 1:14 AM Kevin Hilman <khilman@baylibre.com> wrote:

> Milosz Wasilewski <milosz.wasilewski@linaro.org> writes:
>
> > On Mon, 11 Feb 2019 at 18:27, Kevin Hilman <khilman@baylibre.com> wrote:
> >>
> >> "Milosz Wasilewski" <milosz.wasilewski@linaro.org> writes:
> >>
> >> > On Fri, 8 Feb 2019 at 16:33, "Jan-Simon Möller" <
> JanSimon.Moeller@gmx.de> wrote:
> >> >>
> >> >> Please do not just think of 'building the kernel' - ppl also want to
> test and record/visualize userspace tests (aka custom-built
> kernel+filesystem).
> >> >> Yes, the linux kernel is the main focus for kernelci.org . But
> let's keep the other possible use-cases in mind.
> >> >>
> >> >
> >> > I don't think this is a good idea. Having other use cases in mind
> >> > takes you down a very deep rabbit hole. Creating a general purpose
> >> > result reporting tool is hard. Kernelci reports on kernel results and
> >> > (as I understand) does it right. You can use these tools for other
> >> > purposes but that's a different story.
> >>
> >> Yes, we're focused on kernel-focused testing, but there are multiple
> >> ways to test the kernel, most of which require a combination of kernel +
> >> userspace/distro.
> >>
> >> I think Jan-Simon's point is that as we evolve, while we might focus on
> >> buildroot/debian, we chould not design things in a way that prohibit
> >> using the kernelCI code (and infra) for other types of kernel-focused
> >> testing (e.g. Yocto, other distros, etc.)
> >
> > I guess 'kernel focused' is the key here. I read 'other possible use
> > cases' as a request to build a general purpose reporting tool.
> >
> >>
> >> Stated differently, the current *service* provided by kernelCI.org is
> >> kernel-focused testing.  But, we could (and should, IMO) write the
> >> *software* behind that service in a way that it could be used more
> >> generically if desired.
> >
> > I've been trying to do that for last couple of years with no major
> > success. It might be just me or the fact that it's really hard. Once
> > you try to start reporting on android runtime or openstack things get
> > really complicated.
> >
> > So if you consider running tests that exercise different parts of
> > kernel (like graphics, scheduler...) as 'other use cases' than yes,
> > KCI software should be able to do that. But I don't think going the
> > route of 'general purpose reporting tool' is the right decision.
>
> Agreed.  We should stay focused on *kernel* CI.
>
> But, we don't want to (artificially) limit the types of
> tools/distros/stacks we use to beat up on the kernel.
>

So I think the key point is that root file systems may be coming
from a dynamically generated URL on a per-job basis, and with the
option to not include a ramdisk or a kernel image.  That would
make it possible to replace the regular KernelCI build jobs with
alternative ones that produce a full OS image or a Debian package
or anything more than a plain kernel image, and then feed that
into the reference implementation with LAVA etc...

In fact, I think it would just mean adding some template blocks
around the deploy definitions in the base template.  Then anyone
could crate a "downstream" test plan with alternative URLs using
variables such as the kernel revision to point to a rootfs.

As far as the design document is concerned, it seems that we just
need to state that we don't want to make it hard to achieve that.

On a related topic that has been discussed before, we may want to
use more distros and user-space variants to test the kernel as
part of KernelCI.  Say, to see whether the kernel doesn't hit any
bugs when running an unusual init process or libc implementation
etc...  That should be fine as long as the user-space is a means
of testing a pure upstream kernel revision, so we're not actually
testing the user-space itself or any downstream distro patches.

Guillaume

[-- Attachment #2: Type: text/html, Size: 5400 bytes --]

Re: KernelCI modular pipeline design document

[Kevin Hilman]

Guillaume Tucker <guillaume.tucker@gmail.com> writes:

> On Tue, Feb 12, 2019 at 1:14 AM Kevin Hilman <khilman@baylibre.com> wrote:
>
>> Milosz Wasilewski <milosz.wasilewski@linaro.org> writes:
>>
>> > On Mon, 11 Feb 2019 at 18:27, Kevin Hilman <khilman@baylibre.com> wrote:
>> >>
>> >> "Milosz Wasilewski" <milosz.wasilewski@linaro.org> writes:
>> >>
>> >> > On Fri, 8 Feb 2019 at 16:33, "Jan-Simon Möller" <
>> JanSimon.Moeller@gmx.de> wrote:
>> >> >>
>> >> >> Please do not just think of 'building the kernel' - ppl also want to
>> test and record/visualize userspace tests (aka custom-built
>> kernel+filesystem).
>> >> >> Yes, the linux kernel is the main focus for kernelci.org . But
>> let's keep the other possible use-cases in mind.
>> >> >>
>> >> >
>> >> > I don't think this is a good idea. Having other use cases in mind
>> >> > takes you down a very deep rabbit hole. Creating a general purpose
>> >> > result reporting tool is hard. Kernelci reports on kernel results and
>> >> > (as I understand) does it right. You can use these tools for other
>> >> > purposes but that's a different story.
>> >>
>> >> Yes, we're focused on kernel-focused testing, but there are multiple
>> >> ways to test the kernel, most of which require a combination of kernel +
>> >> userspace/distro.
>> >>
>> >> I think Jan-Simon's point is that as we evolve, while we might focus on
>> >> buildroot/debian, we chould not design things in a way that prohibit
>> >> using the kernelCI code (and infra) for other types of kernel-focused
>> >> testing (e.g. Yocto, other distros, etc.)
>> >
>> > I guess 'kernel focused' is the key here. I read 'other possible use
>> > cases' as a request to build a general purpose reporting tool.
>> >
>> >>
>> >> Stated differently, the current *service* provided by kernelCI.org is
>> >> kernel-focused testing.  But, we could (and should, IMO) write the
>> >> *software* behind that service in a way that it could be used more
>> >> generically if desired.
>> >
>> > I've been trying to do that for last couple of years with no major
>> > success. It might be just me or the fact that it's really hard. Once
>> > you try to start reporting on android runtime or openstack things get
>> > really complicated.
>> >
>> > So if you consider running tests that exercise different parts of
>> > kernel (like graphics, scheduler...) as 'other use cases' than yes,
>> > KCI software should be able to do that. But I don't think going the
>> > route of 'general purpose reporting tool' is the right decision.
>>
>> Agreed.  We should stay focused on *kernel* CI.
>>
>> But, we don't want to (artificially) limit the types of
>> tools/distros/stacks we use to beat up on the kernel.
>>
>
> So I think the key point is that root file systems may be coming
> from a dynamically generated URL on a per-job basis, and with the
> option to not include a ramdisk or a kernel image.  That would
> make it possible to replace the regular KernelCI build jobs with
> alternative ones that produce a full OS image or a Debian package
> or anything more than a plain kernel image, and then feed that
> into the reference implementation with LAVA etc...
>
> In fact, I think it would just mean adding some template blocks
> around the deploy definitions in the base template.  Then anyone
> could crate a "downstream" test plan with alternative URLs using
> variables such as the kernel revision to point to a rootfs.
>
> As far as the design document is concerned, it seems that we just
> need to state that we don't want to make it hard to achieve that.

We probably also need to flesh out a bit more of the metadata fields 
that would help describe a initramfs+rootfs environment so that our
visualization and email reporting can show the basics of what was used.

Kevin