livepatching selftests failure on current master branch

livepatching selftests failure on current master branch

[Miroslav Benes]

Hi,

I noticed that livepatching selftests fail on our master branch 
(https://git.kernel.org/pub/scm/linux/kernel/git/livepatching/livepatching.git/).

...
TEST: busy target module ... not ok

--- expected
+++ result
@@ -7,16 +7,24 @@ livepatch: 'test_klp_callbacks_demo': in
 test_klp_callbacks_demo: pre_patch_callback: vmlinux
 test_klp_callbacks_demo: pre_patch_callback: test_klp_callbacks_busy -> [MODULE_STATE_LIVE] Normal state
 livepatch: 'test_klp_callbacks_demo': starting patching transition
+livepatch: 'test_klp_callbacks_demo': completing patching transition
+test_klp_callbacks_demo: post_patch_callback: vmlinux
+test_klp_callbacks_demo: post_patch_callback: test_klp_callbacks_busy -> [MODULE_STATE_LIVE] Normal state
+livepatch: 'test_klp_callbacks_demo': patching complete
 % modprobe test_klp_callbacks_mod
 livepatch: applying patch 'test_klp_callbacks_demo' to loading module 
'test_klp_callbacks_mod'
 test_klp_callbacks_demo: pre_patch_callback: test_klp_callbacks_mod -> [MODULE_STATE_COMING] Full formed, running module_init
+test_klp_callbacks_demo: post_patch_callback: test_klp_callbacks_mod -> [MODULE_STATE_COMING] Full formed, running module_init
 test_klp_callbacks_mod: test_klp_callbacks_mod_init
 % rmmod test_klp_callbacks_mod
 test_klp_callbacks_mod: test_klp_callbacks_mod_exit
+test_klp_callbacks_demo: pre_unpatch_callback: test_klp_callbacks_mod -> [MODULE_STATE_GOING] Going away
 livepatch: reverting patch 'test_klp_callbacks_demo' on unloading module 
'test_klp_callbacks_mod'
 test_klp_callbacks_demo: post_unpatch_callback: test_klp_callbacks_mod -> [MODULE_STATE_GOING] Going away
 % echo 0 > /sys/kernel/livepatch/test_klp_callbacks_demo/enabled
-livepatch: 'test_klp_callbacks_demo': reversing transition from patching to unpatching
+livepatch: 'test_klp_callbacks_demo': initializing unpatching transition
+test_klp_callbacks_demo: pre_unpatch_callback: vmlinux
+test_klp_callbacks_demo: pre_unpatch_callback: test_klp_callbacks_busy -> [MODULE_STATE_LIVE] Normal state
 livepatch: 'test_klp_callbacks_demo': starting unpatching transition
 livepatch: 'test_klp_callbacks_demo': completing unpatching transition
 test_klp_callbacks_demo: post_unpatch_callback: vmlinux

ERROR: livepatch kselftest(s) failed
not ok 1..2 selftests: livepatch: test-callbacks.sh [FAIL]

which probably means that the consistency model is not in the best shape. 
There were not many livepatch changes in the latest pull request. Stack 
unwinder changes may be connected, so adding Thomas to be aware if it 
leads in this direction.

Unfortunately, I'm leaving in a minute and will be gone till Wednesday, so 
if someone confirms and wants to investigate, definitely feel free to do 
it.

Miroslav

Re: livepatching selftests failure on current master branch

[Joe Lawrence]

On 5/17/19 10:17 AM, Miroslav Benes wrote:
> Hi,
> 
> I noticed that livepatching selftests fail on our master branch
> (https://git.kernel.org/pub/scm/linux/kernel/git/livepatching/livepatching.git/).
> 
> ...
> TEST: busy target module ... not ok
> 
> --- expected
> +++ result
> @@ -7,16 +7,24 @@ livepatch: 'test_klp_callbacks_demo': in
>   test_klp_callbacks_demo: pre_patch_callback: vmlinux
>   test_klp_callbacks_demo: pre_patch_callback: test_klp_callbacks_busy -> [MODULE_STATE_LIVE] Normal state
>   livepatch: 'test_klp_callbacks_demo': starting patching transition
> +livepatch: 'test_klp_callbacks_demo': completing patching transition
> +test_klp_callbacks_demo: post_patch_callback: vmlinux
> +test_klp_callbacks_demo: post_patch_callback: test_klp_callbacks_busy -> [MODULE_STATE_LIVE] Normal state
> +livepatch: 'test_klp_callbacks_demo': patching complete
>   % modprobe test_klp_callbacks_mod
>   livepatch: applying patch 'test_klp_callbacks_demo' to loading module
> 'test_klp_callbacks_mod'
>   test_klp_callbacks_demo: pre_patch_callback: test_klp_callbacks_mod -> [MODULE_STATE_COMING] Full formed, running module_init
> +test_klp_callbacks_demo: post_patch_callback: test_klp_callbacks_mod -> [MODULE_STATE_COMING] Full formed, running module_init
>   test_klp_callbacks_mod: test_klp_callbacks_mod_init
>   % rmmod test_klp_callbacks_mod
>   test_klp_callbacks_mod: test_klp_callbacks_mod_exit
> +test_klp_callbacks_demo: pre_unpatch_callback: test_klp_callbacks_mod -> [MODULE_STATE_GOING] Going away
>   livepatch: reverting patch 'test_klp_callbacks_demo' on unloading module
> 'test_klp_callbacks_mod'
>   test_klp_callbacks_demo: post_unpatch_callback: test_klp_callbacks_mod -> [MODULE_STATE_GOING] Going away
>   % echo 0 > /sys/kernel/livepatch/test_klp_callbacks_demo/enabled
> -livepatch: 'test_klp_callbacks_demo': reversing transition from patching to unpatching
> +livepatch: 'test_klp_callbacks_demo': initializing unpatching transition
> +test_klp_callbacks_demo: pre_unpatch_callback: vmlinux
> +test_klp_callbacks_demo: pre_unpatch_callback: test_klp_callbacks_busy -> [MODULE_STATE_LIVE] Normal state
>   livepatch: 'test_klp_callbacks_demo': starting unpatching transition
>   livepatch: 'test_klp_callbacks_demo': completing unpatching transition
>   test_klp_callbacks_demo: post_unpatch_callback: vmlinux
> 
> ERROR: livepatch kselftest(s) failed
> not ok 1..2 selftests: livepatch: test-callbacks.sh [FAIL]
> 
> which probably means that the consistency model is not in the best shape.
> There were not many livepatch changes in the latest pull request. Stack
> unwinder changes may be connected, so adding Thomas to be aware if it
> leads in this direction.
> 
> Unfortunately, I'm leaving in a minute and will be gone till Wednesday, so
> if someone confirms and wants to investigate, definitely feel free to do
> it.
> 

I will take a look today.  Thanks for reporting.  I hope it's something 
silly in the tests not the consistency model ...

-- Joe

Re: livepatching selftests failure on current master branch

[Joe Lawrence]

[-- Attachment #1: Type: text/plain, Size: 461 bytes --]

On 5/17/19 10:17 AM, Miroslav Benes wrote:
> Hi,
> 
> I noticed that livepatching selftests fail on our master branch
> (https://git.kernel.org/pub/scm/linux/kernel/git/livepatching/livepatching.git/).
> 
> ...

[ adding lkp(a)01.org to this email ]

lkp folks, I was wondering if the kernel selftests were included as part 
of the test-bot and if so, do we need to do anything specific to include 
the livepatching selftests?

Thanks,

-- Joe

Re: livepatching selftests failure on current master branch

[Philip Li]

[-- Attachment #1: Type: text/plain, Size: 1086 bytes --]

On Wed, Jun 05, 2019 at 09:48:02AM -0400, Joe Lawrence wrote:
> On 5/17/19 10:17 AM, Miroslav Benes wrote:
> >Hi,
> >
> >I noticed that livepatching selftests fail on our master branch
> >(https://git.kernel.org/pub/scm/linux/kernel/git/livepatching/livepatching.git/).
> >
> >...
> 
> [ adding lkp(a)01.org to this email ]
> 
> lkp folks, I was wondering if the kernel selftests were included as
> part of the test-bot and if so, do we need to do anything specific
yes, kernel selftest is part of regular execution, which includes the livepatching
test. Also the livepatching.git is under our testing. But we may not successfully
bisect all failures.

The mails related to livepatching test or livepatch  in last two months are

https://lists.01.org/pipermail/lkp/2019-April/010026.html
https://lists.01.org/pipermail/lkp/2019-May/010218.html

Thanks

> to include the livepatching selftests?
> 
> Thanks,
> 
> -- Joe
> _______________________________________________
> LKP mailing list
> LKP(a)lists.01.org
> https://lists.01.org/mailman/listinfo/lkp

Re: livepatching selftests failure on current master branch

[Joe Lawrence]

[-- Attachment #1: Type: text/plain, Size: 1310 bytes --]

On 6/5/19 10:31 AM, Philip Li wrote:
> On Wed, Jun 05, 2019 at 09:48:02AM -0400, Joe Lawrence wrote:
>> On 5/17/19 10:17 AM, Miroslav Benes wrote:
>>> Hi,
>>>
>>> I noticed that livepatching selftests fail on our master branch
>>> (https://git.kernel.org/pub/scm/linux/kernel/git/livepatching/livepatching.git/).
>>>
>>> ...
>>
>> [ adding lkp(a)01.org to this email ]
>>
>> lkp folks, I was wondering if the kernel selftests were included as
>> part of the test-bot and if so, do we need to do anything specific
> yes, kernel selftest is part of regular execution, which includes the livepatching
 > test. Also the livepatching.git is under our testing...

Hi Philip,

Good to hear.  My github tree and Josh's kernel.org tree are included as 
(perhaps other livepatching dev trees as well), so the more coverage the 
better :)

>                                                   ... But we may not successfully
> bisect all failures.

That's okay, though I guess I'm not clear why there wasn't an email 
reporting that the livepatching selftests failed after livepatching.git 
was recently updated to include tglx's stack trace fixes.

Do the tests only run for unique commits (ie, will it skip when 
livepatching.git updates/merges latest linux tree ??)

Thanks,

-- Joe

Re: livepatching selftests failure on current master branch

[Philip Li]

[-- Attachment #1: Type: text/plain, Size: 1607 bytes --]

On Wed, Jun 05, 2019 at 10:51:53AM -0400, Joe Lawrence wrote:
> On 6/5/19 10:31 AM, Philip Li wrote:
> >On Wed, Jun 05, 2019 at 09:48:02AM -0400, Joe Lawrence wrote:
> >>On 5/17/19 10:17 AM, Miroslav Benes wrote:
> >>>Hi,
> >>>
> >>>I noticed that livepatching selftests fail on our master branch
> >>>(https://git.kernel.org/pub/scm/linux/kernel/git/livepatching/livepatching.git/).
> >>>
> >>>...
> >>
> >>[ adding lkp(a)01.org to this email ]
> >>
> >>lkp folks, I was wondering if the kernel selftests were included as
> >>part of the test-bot and if so, do we need to do anything specific
> >yes, kernel selftest is part of regular execution, which includes the livepatching
> > test. Also the livepatching.git is under our testing...
> 
> Hi Philip,
> 
> Good to hear.  My github tree and Josh's kernel.org tree are
> included as (perhaps other livepatching dev trees as well), so the
> more coverage the better :)
> 
> >                                                  ... But we may not successfully
> >bisect all failures.
> 
> That's okay, though I guess I'm not clear why there wasn't an email
> reporting that the livepatching selftests failed after
> livepatching.git was recently updated to include tglx's stack trace
> fixes.
> 
> Do the tests only run for unique commits (ie, will it skip when
> livepatching.git updates/merges latest linux tree ??)
it will not skip, though we are not testing commit by commit. If the issue
is found, and bisect to the bad commit, we will report to author of that
commit for information.

> 
> Thanks,
> 
> -- Joe

Re: livepatching selftests failure on current master branch

[Joe Lawrence]

[-- Attachment #1: Type: text/plain, Size: 1021 bytes --]

On 6/5/19 8:49 PM, Philip Li wrote:
> On Wed, Jun 05, 2019 at 10:51:53AM -0400, Joe Lawrence wrote:
>> Do the tests only run for unique commits (ie, will it skip when
>> livepatching.git updates/merges latest linux tree ??)
> it will not skip, though we are not testing commit by commit. If the issue
> is found, and bisect to the bad commit, we will report to author of that
> commit for information.

Sorry, I'm not following along, let me rephrase my question...

This is the rough timeline that recently occurred:

- changes were made to stacktracing API
-- one commit affected livepatching and broke a selftest
-- Linus pulled the changes
- the livepatching tree updated its master to Linus tree
-- Miroslav noticed that self-tests were broken in the livepatch tree
-- we fixed up the stacktracing API, pushed up to Linus
-- all is good now

Could the 0-day bot testing have helped us spot the self-test breakage 
before Linus pulled or when the livepatching tree updated?

Thanks,

-- Joe

Re: livepatching selftests failure on current master branch

[Philip Li]

[-- Attachment #1: Type: text/plain, Size: 1864 bytes --]

On Thu, Jun 06, 2019 at 02:43:43PM -0400, Joe Lawrence wrote:
> On 6/5/19 8:49 PM, Philip Li wrote:
> >On Wed, Jun 05, 2019 at 10:51:53AM -0400, Joe Lawrence wrote:
> >>Do the tests only run for unique commits (ie, will it skip when
> >>livepatching.git updates/merges latest linux tree ??)
> >it will not skip, though we are not testing commit by commit. If the issue
> >is found, and bisect to the bad commit, we will report to author of that
> >commit for information.
> 
> Sorry, I'm not following along, let me rephrase my question...
sorry for late response, i was taken a few days off.

> 
> This is the rough timeline that recently occurred:
> 
> - changes were made to stacktracing API
> -- one commit affected livepatching and broke a selftest
i assume this is another kernel tree, not necessary the livepatch
kernel tree. This tree may or may not be in our monitor list.
- if not, we can't report is before linus pull.
- if yes, we should be able to detect the failure in normal situation,
and trigger a bisect. But due to how we run the testing internally,
it sometimes can fail to trigger bisect or bisect successfully.

> -- Linus pulled the changes
> - the livepatching tree updated its master to Linus tree
> -- Miroslav noticed that self-tests were broken in the livepatch tree
> -- we fixed up the stacktracing API, pushed up to Linus
> -- all is good now
> 
> Could the 0-day bot testing have helped us spot the self-test
> breakage before Linus pulled or when the livepatching tree updated?
This is similar to previous answer. If the error is not fixed, and exists
in any tree under 0day's monitor, we can figure out the culprit patch.

For now, we will have our best effort to detect regression as many as
possible, while it is not reaching 100% for the test suites we have.

> 
> Thanks,
> 
> -- Joe

Re: livepatching selftests failure on current master branch

[Petr Mladek]

[-- Attachment #1: Type: text/plain, Size: 1013 bytes --]

On Thu 2019-06-06 08:49:43, Philip Li wrote:
> On Wed, Jun 05, 2019 at 10:51:53AM -0400, Joe Lawrence wrote:
> > On 6/5/19 10:31 AM, Philip Li wrote:
> > >On Wed, Jun 05, 2019 at 09:48:02AM -0400, Joe Lawrence wrote:
> > That's okay, though I guess I'm not clear why there wasn't an email
> > reporting that the livepatching selftests failed after
> > livepatching.git was recently updated to include tglx's stack trace
> > fixes.
> > 
> > Do the tests only run for unique commits (ie, will it skip when
> > livepatching.git updates/merges latest linux tree ??)
> it will not skip, though we are not testing commit by commit. If the issue
> is found, and bisect to the bad commit, we will report to author of that
> commit for information.

Bisecting might take a lot of time or even fail. But each selftest
should get associated with a kernel subsystem. It would be helpful
to always inform the subsystem maintainers. Or at least the author
of the selftest that failed.

Best Regards,
Petr

Re: livepatching selftests failure on current master branch

[Philip Li]

[-- Attachment #1: Type: text/plain, Size: 1410 bytes --]

On Tue, Jun 11, 2019 at 09:42:57AM +0200, Petr Mladek wrote:
> On Thu 2019-06-06 08:49:43, Philip Li wrote:
> > On Wed, Jun 05, 2019 at 10:51:53AM -0400, Joe Lawrence wrote:
> > > On 6/5/19 10:31 AM, Philip Li wrote:
> > > >On Wed, Jun 05, 2019 at 09:48:02AM -0400, Joe Lawrence wrote:
> > > That's okay, though I guess I'm not clear why there wasn't an email
> > > reporting that the livepatching selftests failed after
> > > livepatching.git was recently updated to include tglx's stack trace
> > > fixes.
> > > 
> > > Do the tests only run for unique commits (ie, will it skip when
> > > livepatching.git updates/merges latest linux tree ??)
> > it will not skip, though we are not testing commit by commit. If the issue
> > is found, and bisect to the bad commit, we will report to author of that
> > commit for information.
> 
> Bisecting might take a lot of time or even fail. But each selftest
> should get associated with a kernel subsystem. It would be helpful
> to always inform the subsystem maintainers. Or at least the author
> of the selftest that failed.
thanks for the suggestion, we haven't done this yet that we uses merging
approach (merge different repos together to test) to speed up the testing,
so we need to bisect to make sure the issue is reproducible and not related
to merge. But anyway, we will consider this for future todo.

> 
> Best Regards,
> Petr