Automated testing for stable kernel branches?

Automated testing for stable kernel branches?

[Major Hayden]

Hello there,

I am working on a project at Red Hat where we do quick testing on patches for internal kernels before they merge. The goal is to catch bugs or issues before they merge into kernel trees and avoid situations where kernels need time-consuming bisects when lots of patches are merged at once. We aim to put valuable feedback into a kernel developer's inbox within four hours.

Our team has built a pipeline where we merge patches, compile kernels (for various architectures), and run tests on real hardware (various architectures). The current test set is fairly basic and it includes LTP plus some additional open source tests. We are looking to gradually expand those over time as we evaluate which tests provide the most value and find the most problems.

We would love to bring this to upstream kernel repositories and we thought that linux-stable might be a good place to start. The developer/maintainer experience would look something like this:

  1) Developer submits a patchset
  2) Those patches end up in Patchwork
  3) We pull patches from patchwork, compile kernels, and test them
  4) We reply to the thread on the mailing list with a brief set of results (one time per patchset)

Developers do not need to change any existing workflows. We gather the patches, test them, and reply in the appropriate place.

Is this something that the linux-stable community and maintainers would find valuable? If so, feel free to ask any questions about our process and we can go over any of those parts in more detail. If not, please let me know anyway! Our team is always looking for ways to improve. :)

Thanks a bunch for reading this far and I look forward to hearing from you.

--
Major Hayden

Re: Automated testing for stable kernel branches?

[Greg KH]

On Thu, Oct 18, 2018 at 02:24:40PM -0500, Major Hayden wrote:
> Hello there,
> 
> I am working on a project at Red Hat where we do quick testing on
> patches for internal kernels before they merge. The goal is to catch
> bugs or issues before they merge into kernel trees and avoid
> situations where kernels need time-consuming bisects when lots of
> patches are merged at once. We aim to put valuable feedback into a
> kernel developer's inbox within four hours.

Yeah!

> Our team has built a pipeline where we merge patches, compile kernels
> (for various architectures), and run tests on real hardware (various
> architectures). The current test set is fairly basic and it includes
> LTP plus some additional open source tests. We are looking to
> gradually expand those over time as we evaluate which tests provide
> the most value and find the most problems.
> 
> We would love to bring this to upstream kernel repositories and we
> thought that linux-stable might be a good place to start. The
> developer/maintainer experience would look something like this:
> 
>   1) Developer submits a patchset
>   2) Those patches end up in Patchwork
>   3) We pull patches from patchwork, compile kernels, and test them
>   4) We reply to the thread on the mailing list with a brief set of results (one time per patchset)
> 
> Developers do not need to change any existing workflows. We gather the
> patches, test them, and reply in the appropriate place.

Note that not all kernel mailing lists use Patchwork, but I guess you
can always subscribe your own internal copies of it to the lists, right?

> Is this something that the linux-stable community and maintainers
> would find valuable? If so, feel free to ask any questions about our
> process and we can go over any of those parts in more detail. If not,
> please let me know anyway! Our team is always looking for ways to
> improve. :)

You can just go off of my email announcements for the -rc releases and
do testing on that.  That would be a great first step, and if you can
not automatically detect this, I can add you to the email announcement
if you want to trigger off of that.

Also you could watch the linux-stable-rc git tree for updates, that will
be updated at -rc announcement time, and at other "time to take a break"
moments in my development cycle.

Would that work for you?

thanks,

greg k-h

Re: Automated testing for stable kernel branches?

[Major Hayden]

On 10/18/18 2:31 PM, Greg KH wrote:
> Note that not all kernel mailing lists use Patchwork, but I guess you
> can always subscribe your own internal copies of it to the lists, right?

Correct. Luckily, one of my team members is a Patchwork wizard and she can add new lists on our instance.

> You can just go off of my email announcements for the -rc releases and
> do testing on that.  That would be a great first step, and if you can
> not automatically detect this, I can add you to the email announcement
> if you want to trigger off of that.

We can monitor the -rc tags when they are added in the repository if that works for you. I'd like to avoid adding anything else to your already busy workflow. :)

> Also you could watch the linux-stable-rc git tree for updates, that will
> be updated at -rc announcement time, and at other "time to take a break"
> moments in my development cycle.

Would you prefer tests to run prior to the -rc being tagged in linux-stable? We can run tests on a regular schedule there if that's convenient.

Thanks for the quick reply.

--
Major Hayden

Re: Automated testing for stable kernel branches?

[Sudip Mukherjee]

On Thu, Oct 18, 2018 at 8:32 PM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Thu, Oct 18, 2018 at 02:24:40PM -0500, Major Hayden wrote:
> > Hello there,
> >
> > I am working on a project at Red Hat where we do quick testing on
> > patches for internal kernels before they merge. The goal is to catch
> > bugs or issues before they merge into kernel trees and avoid
> > situations where kernels need time-consuming bisects when lots of
> > patches are merged at once. We aim to put valuable feedback into a
> > kernel developer's inbox within four hours.
>
> Yeah!
>

<snip>

> > We would love to bring this to upstream kernel repositories and we
> > thought that linux-stable might be a good place to start. The
> > developer/maintainer experience would look something like this:
> >
> >   1) Developer submits a patchset
> >   2) Those patches end up in Patchwork
> >   3) We pull patches from patchwork, compile kernels, and test them
> >   4) We reply to the thread on the mailing list with a brief set of results (one time per patchset)
> >
> > Developers do not need to change any existing workflows. We gather the
> > patches, test them, and reply in the appropriate place.
>
> Note that not all kernel mailing lists use Patchwork, but I guess you
> can always subscribe your own internal copies of it to the lists, right?

And also, we do not always send patches. We only send patches when it
needs a backport, otherwise we will just mail Greg with the SHA that
needs to be added to the tree.

>
> > Is this something that the linux-stable community and maintainers
> > would find valuable? If so, feel free to ask any questions about our
> > process and we can go over any of those parts in more detail. If not,
> > please let me know anyway! Our team is always looking for ways to
> > improve. :)
>
> You can just go off of my email announcements for the -rc releases and
> do testing on that.  That would be a great first step, and if you can
> not automatically detect this, I can add you to the email announcement
> if you want to trigger off of that.
>
> Also you could watch the linux-stable-rc git tree for updates, that will
> be updated at -rc announcement time, and at other "time to take a break"
> moments in my development cycle.

Or alternatively, you can also monitor the stable-queue repo. I
usually check the stable-queue repo, apply the patches to my personal
tree before doing or testing anything.

-- 
Regards
Sudip

Re: Automated testing for stable kernel branches?

[Major Hayden]

On 10/18/18 3:33 PM, Sudip Mukherjee wrote:
> And also, we do not always send patches. We only send patches when it
> needs a backport, otherwise we will just mail Greg with the SHA that
> needs to be added to the tree.

Thanks, that helps me understand the workflow better.

>> Also you could watch the linux-stable-rc git tree for updates, that will
>> be updated at -rc announcement time, and at other "time to take a break"
>> moments in my development cycle.
> Or alternatively, you can also monitor the stable-queue repo. I
> usually check the stable-queue repo, apply the patches to my personal
> tree before doing or testing anything.

I wasn't aware that one existed! Am I correct to assume this is a holding place for patches which are waiting for inclusion into linux-stable-rc?

Thanks for the quick reply, Sudip.

--
Major Hayden

Re: Automated testing for stable kernel branches?

[Sudip Mukherjee]

On Fri, Oct 19, 2018 at 1:53 PM Major Hayden <major@redhat.com> wrote:
>
> On 10/18/18 3:33 PM, Sudip Mukherjee wrote:
> > And also, we do not always send patches. We only send patches when it
> > needs a backport, otherwise we will just mail Greg with the SHA that
> > needs to be added to the tree.
>
> Thanks, that helps me understand the workflow better.
>
> >> Also you could watch the linux-stable-rc git tree for updates, that will
> >> be updated at -rc announcement time, and at other "time to take a break"
> >> moments in my development cycle.
> > Or alternatively, you can also monitor the stable-queue repo. I
> > usually check the stable-queue repo, apply the patches to my personal
> > tree before doing or testing anything.
>
> I wasn't aware that one existed! Am I correct to assume this is a holding place for patches which are waiting for inclusion into linux-stable-rc?

This is a place where Greg will keep the patches which he accepts for
the stable tree. The stable-rc will finally have everything from this
folder. And the folder will be emptied when the stable tree is
released.
And also, everything will be in patch format.


-- 
Regards
Sudip

Re: Automated testing for stable kernel branches?

[Major Hayden]

On 10/19/18 8:10 AM, Sudip Mukherjee wrote:
> This is a place where Greg will keep the patches which he accepts for
> the stable tree. The stable-rc will finally have everything from this
> folder. And the folder will be emptied when the stable tree is
> released.
> And also, everything will be in patch format.

Perfect. That helps a lot! :)

--
Major Hayden

Re: Automated testing for stable kernel branches?

[Major Hayden]

On 10/18/18 2:31 PM, Greg KH wrote:
>> I am working on a project at Red Hat where we do quick testing on
>> patches for internal kernels before they merge. The goal is to catch
>> bugs or issues before they merge into kernel trees and avoid
>> situations where kernels need time-consuming bisects when lots of
>> patches are merged at once. We aim to put valuable feedback into a
>> kernel developer's inbox within four hours.
>
> Yeah!
> 

One more quick question: When should we send emails?

Should they only be sent when a test fails? Or, should they be sent every time (even for success)?

Thanks!

--
Major Hayden