Automation with 0-day & kdevops

Automation with 0-day & kdevops

[Luis Chamberlain]

Dear 0-day developers,

kdevops [0] has evolved over the years now to a full automation suite
for kernel development and testing. As for the later aspects of it, we
use it to enable complicated subsystem tests such as filesystems
testing. Our automated filesystem coverage has been rather reduced
given the complexity, and so one of its goals was to tackle this. It
also has support to automate testing complex subsystems involving
custom non-upstream yet for things like qemu as well.

While long term we'd like to aim towards automating most of the things
tested under kdevops, it makes sense to start slow with a few simpler
targets. Since kdevops supports kselftests as well, my recommendation
is we start with a few selftests for components we have kernel
maintainers willing to help with either review or help tune up. The
same applies to filesystems. While we have support to test most
popular filesystems it makes sense to start with something simple.

To this end I'd like to see if we can collaborate with 0-day so enable
automation of testing for the following components, the first 3 of
which I help maintain:

With kdevops using its kernel selftests support:

  * Linux kernel modules: using kernel selftests and userspace kmod tests
  * Linux firmware loader: firmware selftests
  * Linux sysctl

As for filesystems I'd like to start with tmpfs as we have a developer
who already has a good baseline for it, and is helping to fix some
fstests bugs found, Daniel Gomez. We also have created different
target profiles to test tmpfs for the different mount options it
supports.

What would this collaboration consist of? Using 0-day's automated to
git clone kdevops, spawn some resouces and run a series of make
commands. If git diff returns non-empty we have a new failure.

[0] https://github.com/linux-kdevops/kdevops

 Luis

Re: Automation with 0-day & kdevops

[Philip Li]

On Fri, Feb 23, 2024 at 07:44:12AM -0800, Luis Chamberlain wrote:
> Dear 0-day developers,
> 
> kdevops [0] has evolved over the years now to a full automation suite
> for kernel development and testing. As for the later aspects of it, we
> use it to enable complicated subsystem tests such as filesystems
> testing. Our automated filesystem coverage has been rather reduced
> given the complexity, and so one of its goals was to tackle this. It
> also has support to automate testing complex subsystems involving
> custom non-upstream yet for things like qemu as well.
> 
> While long term we'd like to aim towards automating most of the things
> tested under kdevops, it makes sense to start slow with a few simpler
> targets. Since kdevops supports kselftests as well, my recommendation
> is we start with a few selftests for components we have kernel
> maintainers willing to help with either review or help tune up. The
> same applies to filesystems. While we have support to test most
> popular filesystems it makes sense to start with something simple.
> 
> To this end I'd like to see if we can collaborate with 0-day so enable
> automation of testing for the following components, the first 3 of
> which I help maintain:
> 
> With kdevops using its kernel selftests support:
> 
>   * Linux kernel modules: using kernel selftests and userspace kmod tests
>   * Linux firmware loader: firmware selftests
>   * Linux sysctl
> 
> As for filesystems I'd like to start with tmpfs as we have a developer
> who already has a good baseline for it, and is helping to fix some
> fstests bugs found, Daniel Gomez. We also have created different
> target profiles to test tmpfs for the different mount options it
> supports.
> 
> What would this collaboration consist of? Using 0-day's automated to
> git clone kdevops, spawn some resouces and run a series of make
> commands. If git diff returns non-empty we have a new failure.

Thanks Luis, we are glad to collaborate on this. We will learn the detail of
kdevops that the website contains a lot of useful materials. And we will update
our progress to you next month for the questions and next steps.

Thanks

> 
> [0] https://github.com/linux-kdevops/kdevops
> 
>  Luis
> 

Re: Automation with 0-day & kdevops

[Luis Chamberlain]

On Sun, Feb 25, 2024 at 07:28:00PM +0800, Philip Li wrote:
> Thanks Luis, we are glad to collaborate on this. We will learn the detail of
> kdevops that the website contains a lot of useful materials. And we will update
> our progress to you next month for the questions and next steps.

Hey Phillip,

I reduced the list to just kdevops@lists.linux.dev

just a heads up on a few things:

  * guestfs is now default -- that should be tried as it is what we
    will build up, and it also has the guestfs/<name>/serial.txt
  * you should try enable the kdevops systemd-remote-journal
  * the kernel selftest baseline -- that needs more love but should be
    easy to complete

We're also discord so we can also discuss there. Happy to also follow up
on zoom with interested developers. Happy to suit any timezone.

  Luis

Re: Automation with 0-day & kdevops

[Philip Li]

On Fri, Mar 15, 2024 at 11:25:16AM -0700, Luis Chamberlain wrote:
> On Sun, Feb 25, 2024 at 07:28:00PM +0800, Philip Li wrote:
> > Thanks Luis, we are glad to collaborate on this. We will learn the detail of
> > kdevops that the website contains a lot of useful materials. And we will update
> > our progress to you next month for the questions and next steps.
> 
> Hey Phillip,
> 
> I reduced the list to just kdevops@lists.linux.dev
> 
> just a heads up on a few things:

Thanks Luis, this month, the 0day side had started to experiment the very basic
kdevops test, such as blktests loop/block, to gather experience. We are still
in the internal discussion to understand the test model and whether the bot can
integrate with it.

The runtime testing in 0day has kind different design and is bisection driven
to identify the bad commit of regression. We are thinking of how to trigger
bisection based on the diff, how to avoid duplication to what 0day is testing, etc.

We will continue the experiment and have the discussion in the mail thread
soon.

> 
>   * guestfs is now default -- that should be tried as it is what we
>     will build up, and it also has the guestfs/<name>/serial.txt
>   * you should try enable the kdevops systemd-remote-journal
>   * the kernel selftest baseline -- that needs more love but should be
>     easy to complete
> 
> We're also discord so we can also discuss there. Happy to also follow up
> on zoom with interested developers. Happy to suit any timezone.
> 
>   Luis