kselftest tree on kernelci.org

kselftest tree on kernelci.org

[Guillaume Tucker]

Hi Shuah,

I've made this PR to start monitoring the "fixes" branch from the
kselftest tree on kernelci.org:

  https://github.com/kernelci/kernelci-core/pull/998

While kselftest changes eventually land in linux-next, monitoring
your tree directly means we can test it earlier and potentially
enable more build variants or experimental tests.  Since
kernelci.org also builds and runs some kselftests we're regularly
finding issues and people are sending fixes for them.  See this
recent story for example:

  https://twitter.com/kernelci/status/1488831497259921409

Keeping an eye on kselftest patches with kernelci.org means we
can verify that fixes do what they're supposed to do with a much
larger test coverage than what individual developers can do.
We've been applying kselftest fixes on a branch managed by
kernelci.org to verify them in the past, but having the actual
kselftest tree part of the workflow would seem much better.

There are several branches in your tree, while "fixes" seemed
like the most useful one to pick I see there is also a "kernelci"
branch too but it hasn't been updated for a while, reviving it
could give you the possibility to test patches through
kernelci.org before applying them on other branches that get
pulled into linux-next and mainline.

Many things could potentially be done with arbitrary builds and
tests etc.  These are some initial suggestions.  How does this
sound?

Best wishes,
Guillaume

Re: kselftest tree on kernelci.org

[Shuah Khan]

Hi Guillaume,

On 2/2/22 6:32 AM, Guillaume Tucker wrote:
> Hi Shuah,
> 
> I've made this PR to start monitoring the "fixes" branch from the
> kselftest tree on kernelci.org:
> 
>    https://github.com/kernelci/kernelci-core/pull/998
> 

Thank you.

> While kselftest changes eventually land in linux-next, monitoring
> your tree directly means we can test it earlier and potentially
> enable more build variants or experimental tests.  Since
> kernelci.org also builds and runs some kselftests we're regularly
> finding issues and people are sending fixes for them.  See this
> recent story for example:
> 
>    https://twitter.com/kernelci/status/1488831497259921409
> 
> Keeping an eye on kselftest patches with kernelci.org means we
> can verify that fixes do what they're supposed to do with a much
> larger test coverage than what individual developers can do.
> We've been applying kselftest fixes on a branch managed by
> kernelci.org to verify them in the past, but having the actual
> kselftest tree part of the workflow would seem much better.
> 

Absolutely.

> There are several branches in your tree, while "fixes" seemed
> like the most useful one to pick I see there is also a "kernelci"
> branch too but it hasn't been updated for a while, reviving it
> could give you the possibility to test patches through
> kernelci.org before applying them on other branches that get
> pulled into linux-next and mainline.
> 

This branch was a topic branch specific for changes I made for
kernelci runs to be cleaner - I should delete this.

If you are looking for other branches to monitor in addition to
"fixes" branch, the following are the ones to add:

next (for the merge window), kunit (kunit changes slated for merge window),
kunit-fixes

> Many things could potentially be done with arbitrary builds and
> tests etc.  These are some initial suggestions.  How does this
> sound?

Sounds great to me. Since selftest patches flow through various
subsystem trees, having kernelci keep an eye is great. This would
be another pair of eyes in addition to occasional tests I run and
Linaro (LKTP) monitoring next.

How often do you send reports - I will watch out for them. Thanks
again for taking the initiative to add kselftest to kernelci.

thanks,
-- Shuah

Re: kselftest tree on kernelci.org

[Guillaume Tucker]

On 02/02/2022 15:23, Shuah Khan wrote:
> Hi Guillaume,
> 
> On 2/2/22 6:32 AM, Guillaume Tucker wrote:
>> Hi Shuah,
>>
>> I've made this PR to start monitoring the "fixes" branch from the
>> kselftest tree on kernelci.org:
>>
>>    https://github.com/kernelci/kernelci-core/pull/998
>>
> 
> Thank you.

You're welcome.

>> While kselftest changes eventually land in linux-next, monitoring
>> your tree directly means we can test it earlier and potentially
>> enable more build variants or experimental tests.  Since
>> kernelci.org also builds and runs some kselftests we're regularly
>> finding issues and people are sending fixes for them.  See this
>> recent story for example:
>>
>>    https://twitter.com/kernelci/status/1488831497259921409
>>
>> Keeping an eye on kselftest patches with kernelci.org means we
>> can verify that fixes do what they're supposed to do with a much
>> larger test coverage than what individual developers can do.
>> We've been applying kselftest fixes on a branch managed by
>> kernelci.org to verify them in the past, but having the actual
>> kselftest tree part of the workflow would seem much better.
>>
> 
> Absolutely.
> 
>> There are several branches in your tree, while "fixes" seemed
>> like the most useful one to pick I see there is also a "kernelci"
>> branch too but it hasn't been updated for a while, reviving it
>> could give you the possibility to test patches through
>> kernelci.org before applying them on other branches that get
>> pulled into linux-next and mainline.
>>
> 
> This branch was a topic branch specific for changes I made for
> kernelci runs to be cleaner - I should delete this.
> 
> If you are looking for other branches to monitor in addition to
> "fixes" branch, the following are the ones to add:
> 
> next (for the merge window), kunit (kunit changes slated for merge window),
> kunit-fixes

I see these 4 branches (fixes, next, kunit, kunit-fixes) are all
merged into linux-next.  So it seems like the best thing to do
would be to cover them with a very lightweight number of builds and
tests focused on what they are about: only run kselftest on the
fixes and next branches, and only KUnit on kunit and kunit-fixes.
At the moment, KUnit is not run by kernelci.org (coming soon) so I
think we could just add the next branch for kselftest.  Then
linux-next will be tested with maximum coverage anyway so if
something subtle gets missed with the early tests it should get
caught the following day at the latest with linux-next.

>> Many things could potentially be done with arbitrary builds and
>> tests etc.  These are some initial suggestions.  How does this
>> sound?
> 
> Sounds great to me. Since selftest patches flow through various
> subsystem trees, having kernelci keep an eye is great. This would
> be another pair of eyes in addition to occasional tests I run and
> Linaro (LKTP) monitoring next.
> 
> How often do you send reports - I will watch out for them. Thanks
> again for taking the initiative to add kselftest to kernelci.

Builds and tests are run every time a new revision is detected on
the branches being monitored.  Then when they complete, a report
is sent.  It can take a while, but with a small number of builds
you could get results within an hour.  We could get the reports
sent to the linux-kselftest mailing list and your own address if
you want.

Also this configuration is all under source control on GitHub so
feel free to make changes to it in the future as you see fit.

Best wishes,
Guillaume

Re: kselftest tree on kernelci.org

[Shuah Khan]

On 2/3/22 11:19 AM, Guillaume Tucker wrote:
> On 02/02/2022 15:23, Shuah Khan wrote:
>> Hi Guillaume,
>>

> 
> I see these 4 branches (fixes, next, kunit, kunit-fixes) are all
> merged into linux-next.  So it seems like the best thing to do
> would be to cover them with a very lightweight number of builds and
> tests focused on what they are about: only run kselftest on the
> fixes and next branches, and only KUnit on kunit and kunit-fixes.
> At the moment, KUnit is not run by kernelci.org (coming soon) so I
> think we could just add the next branch for kselftest.  Then
> linux-next will be tested with maximum coverage anyway so if
> something subtle gets missed with the early tests it should get
> caught the following day at the latest with linux-next.
> 
>>> Many things could potentially be done with arbitrary builds and
>>> tests etc.  These are some initial suggestions.  How does this
>>> sound?
>>

Sounds good to me. The things that tend to break are:

- test builds at times due to seemingly innocuous changes to fix
   static analysis warnings. build test is good for catching these
-

>> Sounds great to me. Since selftest patches flow through various
>> subsystem trees, having kernelci keep an eye is great. This would
>> be another pair of eyes in addition to occasional tests I run and
>> Linaro (LKTP) monitoring next.
>>
>> How often do you send reports - I will watch out for them. Thanks
>> again for taking the initiative to add kselftest to kernelci.
> 
> Builds and tests are run every time a new revision is detected on
> the branches being monitored.  Then when they complete, a report
> is sent.  It can take a while, but with a small number of builds
> you could get results within an hour.  We could get the reports
> sent to the linux-kselftest mailing list and your own address if
> you want.
> 

Please send it to my email address as well.

> Also this configuration is all under source control on GitHub so
> feel free to make changes to it in the future as you see fit.
> 

Will do.

thanks,
-- Shuah