kdevops BoF at LSFMM

kdevops BoF at LSFMM

[Luis Chamberlain]

Dear LPC session leads,

We'd like to gather together and talk about current ongoing
developments / changes on kdevops at LSFMM. Those interested in
automation on complex workflows with kdevops are also welcomed. This
is best addressed informally, but since I see an open slot for at
10:30am for Tuesday, figured I'd check to see if we can snatch it.
BoFs for filesystems are scheduled towards the end of the conference
on Wednesday it seems, so ideally this would just take place then, but
the last BoF for XFS at Linux Plumbers took... 4 hours, and if such
filesystem BoFs take place I suspect each FS developer would also want
to attend their own respective FS BoF... so perhaps best we get a
kdevops BoF out of the way before the respective filesystem BoFs.

Agenda items?

Guestfs migration progress - have we killed vagrant?
Automation on testing filesystem baselines
xarray / maple tree testing and userspace testing
OpenTofu
kdevops-results-archive split

Any others?

 Luis

Re: kdevops BoF at LSFMM

[Chuck Lever III]

> On May 7, 2024, at 2:44 PM, Luis Chamberlain <mcgrof@kernel.org> wrote:
> 
> Dear LPC session leads,
> 
> We'd like to gather together and talk about current ongoing
> developments / changes on kdevops at LSFMM. Those interested in
> automation on complex workflows with kdevops are also welcomed. This
> is best addressed informally, but since I see an open slot for at
> 10:30am for Tuesday, figured I'd check to see if we can snatch it.
> BoFs for filesystems are scheduled towards the end of the conference
> on Wednesday it seems, so ideally this would just take place then, but
> the last BoF for XFS at Linux Plumbers took... 4 hours, and if such
> filesystem BoFs take place I suspect each FS developer would also want
> to attend their own respective FS BoF... so perhaps best we get a
> kdevops BoF out of the way before the respective filesystem BoFs.
> 
> Agenda items?
> 
> Guestfs migration progress - have we killed vagrant?
> Automation on testing filesystem baselines
> xarray / maple tree testing and userspace testing
> OpenTofu
> kdevops-results-archive split

results archive - would be nice to have utilities for
workflows (not fstests) to manage their results.

We've added robust NFS, SMB, and tmpfs support over
the past year. What about 9p? bcachefs? Others?

I'm about to add support for creating iSCSI LUNs on
demand. What about NVMeoF as file system backing store?


--
Chuck Lever

Re: kdevops BoF at LSFMM

[Amir Goldstein]

On Tue, May 7, 2024 at 9:44 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> Dear LPC session leads,
>
> We'd like to gather together and talk about current ongoing
> developments / changes on kdevops at LSFMM. Those interested in
> automation on complex workflows with kdevops are also welcomed. This
> is best addressed informally, but since I see an open slot for at
> 10:30am for Tuesday, figured I'd check to see if we can snatch it.

The empty slot is there for flexibility of the schedule and also
wouldn't storage/MM people be interested in kdevops?

I've placed you session instead of the FS lightning talks on Tuesday
after Leah's FS testing session.
There are enough slots for FS lightning talks.

There are several empty slots throughout the agenda left for
flexibility, including the one you mentioned on Tue morning.
kdevops session is for a very specialized group of developers,
so if that group is assembled and decides to use an earlier slot
we can do that on the spot.

> BoFs for filesystems are scheduled towards the end of the conference
> on Wednesday it seems, so ideally this would just take place then, but
> the last BoF for XFS at Linux Plumbers took... 4 hours, and if such
> filesystem BoFs take place I suspect each FS developer would also want
> to attend their own respective FS BoF...

I certainly hope that FS maintainers will use this gathering to have
their own respective BoFs.
If anyone would like to lead an FS BoF in the FS track room,
please let me know and I will place it on the agenda.

Thanks,
Amir.

Re: kdevops BoF at LSFMM

[Luis Chamberlain]

On Wed, May 08, 2024 at 10:45:33AM +0300, Amir Goldstein wrote:
> On Tue, May 7, 2024 at 9:44 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> >
> > Dear LPC session leads,
> >
> > We'd like to gather together and talk about current ongoing
> > developments / changes on kdevops at LSFMM. Those interested in
> > automation on complex workflows with kdevops are also welcomed. This
> > is best addressed informally
> 
> wouldn't storage/MM people be interested in kdevops?

It is up to them, I mean, I've started to work on mmtests integration
so we can help test memory fragmentation, and plan is to integrate
automation of maple tree and xarray shortly, mm folks are more than
welcomed!

> I've placed you session instead of the FS lightning talks on Tuesday
> after Leah's FS testing session.
> There are enough slots for FS lightning talks.

That works great, thanks!

> kdevops session is for a very specialized group of developers,
> so if that group is assembled and decides to use an earlier slot
> we can do that on the spot.

I think the current timing is perfect, and does not even conflict with
mm folks, if they want to join.

  Luis

Re: kdevops BoF at LSFMM

[Matthew Wilcox]

On Wed, May 08, 2024 at 08:57:16AM -0700, Luis Chamberlain wrote:
> It is up to them, I mean, I've started to work on mmtests integration
> so we can help test memory fragmentation, and plan is to integrate
> automation of maple tree and xarray shortly, mm folks are more than
> welcomed!

I don't see how kdevops integration of the radix tree, xarray and maple
tree test suite is useful.  It's a white-box test suite.  kdevops is
more suited for black-box testing like xfstests and mmtests.

When we find a problem in the various data structures through the black
box testing of their users (like the page cache or the VMA), it's on us
to add tests that exercise that functionality.  Like 2a0774c2886d.

The important thing is that the data structures by design don't depend
on much of the kernel.  So if they work in-and-of themselves, there's
not much to gain by doing tests in-kernel.

Re: kdevops BoF at LSFMM

[Steve French]

On Wed, May 8, 2024 at 2:48 AM Amir Goldstein <amir73il@gmail.com> wrote:
>
> On Tue, May 7, 2024 at 9:44 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
> >
> > Dear LPC session leads,
> >
> > We'd like to gather together and talk about current ongoing
> > developments / changes on kdevops at LSFMM. Those interested in
> > automation on complex workflows with kdevops are also welcomed. This
> > is best addressed informally, but since I see an open slot for at
> > 10:30am for Tuesday, figured I'd check to see if we can snatch it.
>
> The empty slot is there for flexibility of the schedule and also
> wouldn't storage/MM people be interested in kdevops?
>
> I've placed you session instead of the FS lightning talks on Tuesday
> after Leah's FS testing session.
> There are enough slots for FS lightning talks.
>
> There are several empty slots throughout the agenda left for
> flexibility, including the one you mentioned on Tue morning.
> kdevops session is for a very specialized group of developers,
> so if that group is assembled and decides to use an earlier slot
> we can do that on the spot.

kdevops could be *extrememly* useful to understand better (and
to share "best practices" and ideas on testing from various filesystems)

I would be very happy if there were an easy way to do three things
faster/easier:
1) make it easier to run a reasonably large set of fs tests automatically
on checkin of a commit or set of commits (e.g. to an externally visible
github branch) before it goes in linux-next, and a larger set
of test automation that is automatically run on P/Rs (I kick these tests
off semi-manually for cifs.ko and ksmbd.ko today)
2) make it easier as a maintainer to get reports of automated testing of
stable-rc (or automate running of tests against stable-rc by filesystem type
and send failures to the specific fs's mailing list).  Make the tests run
for a particular fs more visible, so maintainers/contributors can note
where important tests are left out against a particular fs
3) make it easier to auto-bisect what commit regressed when a failing test
is spotted
4) make it easier to automatically enable certain fs specific debug
tooling (e.g. eBPF scripts or trace points or log capturing) when a
test fails, or when a test fails - enable tracing and restart tests
5) make it easier to collect log output at the end of each test to
catch "suspicious" things (like network reconnects/timeouts, dmesg
events logged, fs specific stats or debug data that show excessive
failures or slow responses)
6) an easy way to tell if a kdevops run is "suspiciously slow" (ie if a test
or set of tests is more than 20% slower than the baseline test run, it
could indicate a performance regression that needs to be bisected
and identified)

-- 
Thanks,

Steve

Re: kdevops BoF at LSFMM

[Chuck Lever III]

> On May 8, 2024, at 1:45 PM, Steve French <smfrench@gmail.com> wrote:
> 
> I would be very happy if there were an easy way to do three things
> faster/easier:
> 1) make it easier to run a reasonably large set of fs tests automatically
> on checkin of a commit or set of commits (e.g. to an externally visible
> github branch) before it goes in linux-next, and a larger set
> of test automation that is automatically run on P/Rs (I kick these tests
> off semi-manually for cifs.ko and ksmbd.ko today)
> 2) make it easier as a maintainer to get reports of automated testing of
> stable-rc (or automate running of tests against stable-rc by filesystem type
> and send failures to the specific fs's mailing list).  Make the tests run
> for a particular fs more visible, so maintainers/contributors can note
> where important tests are left out against a particular fs

In my experience, these require the addition of a CI
apparatus like BuildBot or Jenkins -- they are not
directly part of kdevops' mission. Scott Mayhew and
I have been playing with BuildBot, and there are some
areas where integration between kdevops and BuildBot
could be improved (and could be discussed next week).


> 3) make it easier to auto-bisect what commit regressed when a failing test
> is spotted

Jeff Layton has mentioned this as well. I don't think
it would be impossible to get kdevops to orchestrate
a bisect, as long as it has an automatic way to decide
when to use "git bisect {good|bad}"


> 6) an easy way to tell if a kdevops run is "suspiciously slow" (ie if a test
> or set of tests is more than 20% slower than the baseline test run, it
> could indicate a performance regression that needs to be bisected
> and identified)

Well sometimes things are just slow because you've built
a test kernel with KASAN and lockdep, or because there are
other jobs running on your test system. Also, due to all
the virtualization involved, it might be difficult to get
consistent performance measurements.

This one seems like it would be hard to engineer, but maybe
there's something that could be done?


--
Chuck Lever

Re: kdevops BoF at LSFMM

[Luis Chamberlain]

On Wed, May 08, 2024 at 05:54:50PM +0000, Chuck Lever III wrote:
> 
> 
> > On May 8, 2024, at 1:45 PM, Steve French <smfrench@gmail.com> wrote:
> > 
> > I would be very happy if there were an easy way to do three things
> > faster/easier:
> > 1) make it easier to run a reasonably large set of fs tests automatically
> > on checkin of a commit or set of commits (e.g. to an externally visible
> > github branch) before it goes in linux-next, and a larger set
> > of test automation that is automatically run on P/Rs (I kick these tests
> > off semi-manually for cifs.ko and ksmbd.ko today)
> > 2) make it easier as a maintainer to get reports of automated testing of
> > stable-rc (or automate running of tests against stable-rc by filesystem type
> > and send failures to the specific fs's mailing list).  Make the tests run
> > for a particular fs more visible, so maintainers/contributors can note
> > where important tests are left out against a particular fs
> 
> In my experience, these require the addition of a CI
> apparatus like BuildBot or Jenkins -- they are not
> directly part of kdevops' mission.

Song Liu and Paul E Luse will have a talk on Wednesday about using a
CI framework for md/raid. The holy grail I think here is that they
have used their experience with eBPF patchwork CI integration, and
I think everyone likely wants something similar:

https://patchwork.kernel.org/project/netdevbpf/list/

The S / W / F is Success / warning/ fail.

I'd like to see how we can do that for kdevops. The work is already
put in place to ramp up complex workflows, now we just have to launch
them and collect information.

> Scott Mayhew and
> I have been playing with BuildBot, and there are some
> areas where integration between kdevops and BuildBot
> could be improved (and could be discussed next week).

Neat!

> > 3) make it easier to auto-bisect what commit regressed when a failing test
> > is spotted
> 
> Jeff Layton has mentioned this as well. I don't think
> it would be impossible to get kdevops to orchestrate
> a bisect, as long as it has an automatic way to decide
> when to use "git bisect {good|bad}"

Kent alreeady seems to have this working too, we should try to see what
we can leverage.

  Luis

Re: kdevops BoF at LSFMM

[Theodore Ts'o]

On Wed, May 08, 2024 at 05:54:50PM +0000, Chuck Lever III wrote:
> 
> In my experience, these require the addition of a CI
> apparatus like BuildBot or Jenkins -- they are not
> directly part of kdevops' mission. Scott Mayhew and
> I have been playing with BuildBot, and there are some
> areas where integration between kdevops and BuildBot
> could be improved (and could be discussed next week).

This is support which gce-xfstests has.  You can either have it watch
a particular branch on a git repo, and whenever it changes it will
kick off a tests:

   gce-xfstests ltm [-c <cfg>] [-g <group>]|[<tests>] ... [--repo <url>] \
      --watch <branch>

As an example:

   gce-xfstests ltm -c ext4/all -g auto --repo ext4 --watch dev

Or you can specify a specific commit that you want to run tests on:

   gce-xfstests ltm -c ext4/all -g auto --repo ext4 --commit dev

In the two examples above ext4 is an abbrevation so you don't have to
type the full URL; you can define additional abbrevs in your config
file.

> > 3) make it easier to auto-bisect what commit regressed when a failing test
> > is spotted
> 
> Jeff Layton has mentioned this as well. I don't think
> it would be impossible to get kdevops to orchestrate
> a bisect, as long as it has an automatic way to decide
> when to use "git bisect {good|bad}"

gce-xfstests has this as well:

    gce-xfstests ltm [-c <cfg>] [-g <group>]|[<tests>] ... [--repo <url>] \
        --bisect-bad <bad_rev> --bisect-good <good_rev> \
        [--bisect-good <good_rev1> ...]
    
So yeah, not that hard --- we had intern learn how to program in Go
and to implement the base lightweight test manager, and a team of
undergraduates who implemented the build server, test spinner, and
auto-bisector as part of their software engineering classes's project.  :-)

The code is up on github at github.com/tytso/xfstests-bld if anyone
wants to repurpose it.

	      	      	       		- Ted