Re: [PATCH v2 0/2] livepatch: Move tests from lib/livepatch to selftests/livepatch

[Shuah Khan <skhan@linuxfoundation.org>]

On 11/30/22 15:22, Joe Lawrence wrote:
> On 7/15/22 10:45 AM, Petr Mladek wrote:
>> On Fri 2022-07-01 16:13:50, Shuah Khan wrote:
>>> On 7/1/22 1:48 AM, Miroslav Benes wrote:
>>>> On Thu, 30 Jun 2022, Shuah Khan wrote:
>>>>>
>>>>> Sorry Nack on this. Let's not add modules under selftests. Any usage of
>>>>> module_init()
>>>>> doesn't belong under selftests.
>>>
>>> Yes I did and after reviewing and thinking about it some more, I decided this
>>> is the right direction go down on.
>>
>> Do you have some particular reason why building modules in selftests
>> directory might cause problems, please?
>>

My reasons are that with this change module_init() propagates out of
strictly kernel space and now is in selftests which are user-space.
Any changes to this interface will be tied to user-space change.

This is my main concern. That is reason why I still ask the question
about why is it necessary to make this change other than self-contained
sources?

>> IMHO, the reason that the test modules are in lib is because the
>> modules were there before selftests. Developers historically loaded them
>> manually or they were built-in. Selftest were added later and are just
>> another way how the module can be loaded. This is the case,
>> for example, for lib/test_printf.c.
>>
>> Otherwise, I do not see any big difference between building binaries
>> and modules under tools/tests/selftests. As I said, in the older
>> thread, IMHO, it makes more sense to have the selftest sources
>> self-contained.

Modules under lib are built when kernel gets built as opposed to when
tests are built. So there is the difference in build order. I do see
a difference from that point of view.

Yes, moving modules under selftests does make the tests self contained.

>>
>>
>> There actually seems to be a principal problem in the following use
>> case:
>>
>> --- cut Documentation/dev-tools/kselftest.rst ---
>> Kselftest from mainline can be run on older stable kernels. Running tests
>> from mainline offers the best coverage. Several test rings run mainline
>> kselftest suite on stable releases. The reason is that when a new test
>> gets added to test existing code to regression test a bug, we should be
>> able to run that test on an older kernel. Hence, it is important to keep
>> code that can still test an older kernel and make sure it skips the test
>> gracefully on newer releases.
>> --- cut Documentation/dev-tools/kselftest.rst ---
>>
>> together with
>>
>> --- cut Documentation/dev-tools/kselftest.rst ---
>>   * First use the headers inside the kernel source and/or git repo, and then the
>>     system headers.  Headers for the kernel release as opposed to headers
>>     installed by the distro on the system should be the primary focus to be able
>>     to find regressions.
>> --- cut Documentation/dev-tools/kselftest.rst ---
>>
>> It means that selftests should support running binaries built against
>> newer kernel sources on system running older kernel. But this might
>> be pretty hard to achieve and maintain.
>>
>> The normal kernel rules are exactly the opposite. Old binaries must
>> be able to run on newer kernels. The old binaries were built against
>> older headers.
>>

This case is applicable to when tests are built on a development system
and binaries are moved to run on a target system.

In general, newer tests offer the best coverage, hence the recommendation
to run newer tests on older kernels assuming that the tests are built
on a newer kernel and backwards should run in a backwards compatible
way on older kernels.

Your use-case might be different from this where you do build tests
on older kernels and run them on it in which case, you might have a
requirement to revision match the tests and kernel. You can still do
so.

>> IMHO, the testing of stable kernels makes perfect sense. And if we
>> want to support it seriously than we need to allow building new
>> selftests against headers from the old to-be-tested kernel. And
>> it will be possible only when the selftests sources are as much
>> selfcontained as possible.
>>

Do you have a requirement that livepatch test has to be revision
matched with the kernel? Even if that is the case, there is no real
reason to move modules under selftests other than keeping them in
one location.

Also I didn't see any changes to README that explains this move and
that this change now makes this test now depends on kernel only
interfaces and hence will have to follow modules built outside of
kernel build.

>> Does this makes any sense, please?
>>
> 
> Gentle bump.  Shuah, I believe that Marcos will be preparing a v3 based
> on review comments on the second patch.  We never resolved questions
> surrounding building modules selftests/ (the first patch) though.
> 

You can send patches again and I would like to hear good reasons other
than self-containing the sources.

thanks,
-- Shuah