Converting kselftest test modules to kunit

Converting kselftest test modules to kunit

[Muhammad Usama Anjum]

Hi Kees and All,

There are several tests in kselftest subsystem which load modules to tests
the internals of the kernel. Most of these test modules are just loaded by
the kselftest, their status isn't read and reported to the user logs. Hence
they don't provide benefit of executing those tests.

I've found patches from Kees where he has been converting such kselftests
to kunit tests [1]. The probable motivation is to move tests output of
kselftest subsystem which only triggers tests without correctly reporting
the results. On the other hand, kunit is there to test the kernel's
internal functions which can't be done by userspace.

Kselftest:	Test user facing APIs from userspace
Kunit:		Test kernel's internal functions from kernelspace

This brings me to conclusion that kselftest which are loading modules to
test kernelspace should be converted to kunit tests. I've noted several
such kselftests.

This is just my understanding. Please mention if I'm correct above or more
reasons to support kselftest test modules transformation into kunit test.

[1] https://lore.kernel.org/all/20221018082824.never.845-kees@kernel.org/

-- 
BR,
Muhammad Usama Anjum

Re: Converting kselftest test modules to kunit

[Kees Cook]

On Mon, Jul 15, 2024 at 03:09:24PM +0500, Muhammad Usama Anjum wrote:
> Hi Kees and All,
> 
> There are several tests in kselftest subsystem which load modules to tests
> the internals of the kernel. Most of these test modules are just loaded by
> the kselftest, their status isn't read and reported to the user logs. Hence
> they don't provide benefit of executing those tests.
> 
> I've found patches from Kees where he has been converting such kselftests
> to kunit tests [1]. The probable motivation is to move tests output of
> kselftest subsystem which only triggers tests without correctly reporting
> the results. On the other hand, kunit is there to test the kernel's
> internal functions which can't be done by userspace.
> 
> Kselftest:	Test user facing APIs from userspace
> Kunit:		Test kernel's internal functions from kernelspace

I would say this is a reasonable guide to how these things should
be separated, yes. That said, much of what was kind of ad-hoc kernel
internals testing that was triggered via kselftests is better done via
KUnit these days, but not everything.

> This brings me to conclusion that kselftest which are loading modules to
> test kernelspace should be converted to kunit tests. I've noted several
> such kselftests.

I would tend to agree, yes. Which stand out to you? I've mainly been
doing the conversions when I find myself wanting to add new tests, etc.

-- 
Kees Cook

Re: Converting kselftest test modules to kunit

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 1479 bytes --]

On Mon, 15 Jul 2024 at 18:09, Muhammad Usama Anjum
<usama.anjum@collabora.com> wrote:
>
> Hi Kees and All,
>
> There are several tests in kselftest subsystem which load modules to tests
> the internals of the kernel. Most of these test modules are just loaded by
> the kselftest, their status isn't read and reported to the user logs. Hence
> they don't provide benefit of executing those tests.
>
> I've found patches from Kees where he has been converting such kselftests
> to kunit tests [1]. The probable motivation is to move tests output of
> kselftest subsystem which only triggers tests without correctly reporting
> the results. On the other hand, kunit is there to test the kernel's
> internal functions which can't be done by userspace.
>
> Kselftest:      Test user facing APIs from userspace
> Kunit:          Test kernel's internal functions from kernelspace

Yes: this is how we'd like to split things up. There are still a few
cases where you might want to use kselftest to test something other
than a user-facing API (if you needed to set up some complicated
userspace structures, etc), or cases where KUnit might be used to test
something other than individual pieces of functionality, but that
categorisation is a good start.

The Documentation/dev-tools/testing-overview.rst page has a more
detailed look at when to use which test framework (which basically
just repeats those rules):
https://docs.kernel.org/dev-tools/testing-overview.html

Cheers,
-- David

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4014 bytes --]

Re: Converting kselftest test modules to kunit

[Muhammad Usama Anjum]

On 7/15/24 9:40 PM, Kees Cook wrote:
> On Mon, Jul 15, 2024 at 03:09:24PM +0500, Muhammad Usama Anjum wrote:
>> Hi Kees and All,
>>
>> There are several tests in kselftest subsystem which load modules to tests
>> the internals of the kernel. Most of these test modules are just loaded by
>> the kselftest, their status isn't read and reported to the user logs. Hence
>> they don't provide benefit of executing those tests.
>>
>> I've found patches from Kees where he has been converting such kselftests
>> to kunit tests [1]. The probable motivation is to move tests output of
>> kselftest subsystem which only triggers tests without correctly reporting
>> the results. On the other hand, kunit is there to test the kernel's
>> internal functions which can't be done by userspace.
>>
>> Kselftest:	Test user facing APIs from userspace
>> Kunit:		Test kernel's internal functions from kernelspace
> 
> I would say this is a reasonable guide to how these things should
> be separated, yes. That said, much of what was kind of ad-hoc kernel
> internals testing that was triggered via kselftests is better done via
> KUnit these days, but not everything.
I started investigated when I found that kselftest doesn't parse the kernel
logs to mark these tests pass/fail. (kselftest/lib is good example of it)

> 
>> This brings me to conclusion that kselftest which are loading modules to
>> test kernelspace should be converted to kunit tests. I've noted several
>> such kselftests.
> 
> I would tend to agree, yes. Which stand out to you? I've mainly been
> doing the conversions when I find myself wanting to add new tests, etc.
lib
	test_bitmap
	prime_numbers
	test_printf
	test_scanf
	test_strscpy (already converted, need to remove this test)
lock
	test-ww_mutex module
net
	test_blackhole_dev
user
	test_user_copy (probably already converted, need to remove this test)
firmware
	test_firmware
fpu
	test_fpu

Most of these modules are found in lib/*.

Would it be desired to move these to kunit?

-- 
BR,
Muhammad Usama Anjum

Re: Converting kselftest test modules to kunit

[Kees Cook]

On Tue, Jul 16, 2024 at 01:11:14PM +0500, Muhammad Usama Anjum wrote:
> On 7/15/24 9:40 PM, Kees Cook wrote:
> > On Mon, Jul 15, 2024 at 03:09:24PM +0500, Muhammad Usama Anjum wrote:
> >> Hi Kees and All,
> >>
> >> There are several tests in kselftest subsystem which load modules to tests
> >> the internals of the kernel. Most of these test modules are just loaded by
> >> the kselftest, their status isn't read and reported to the user logs. Hence
> >> they don't provide benefit of executing those tests.
> >>
> >> I've found patches from Kees where he has been converting such kselftests
> >> to kunit tests [1]. The probable motivation is to move tests output of
> >> kselftest subsystem which only triggers tests without correctly reporting
> >> the results. On the other hand, kunit is there to test the kernel's
> >> internal functions which can't be done by userspace.
> >>
> >> Kselftest:	Test user facing APIs from userspace
> >> Kunit:		Test kernel's internal functions from kernelspace
> > 
> > I would say this is a reasonable guide to how these things should
> > be separated, yes. That said, much of what was kind of ad-hoc kernel
> > internals testing that was triggered via kselftests is better done via
> > KUnit these days, but not everything.
> I started investigated when I found that kselftest doesn't parse the kernel
> logs to mark these tests pass/fail. (kselftest/lib is good example of it)
> 
> > 
> >> This brings me to conclusion that kselftest which are loading modules to
> >> test kernelspace should be converted to kunit tests. I've noted several
> >> such kselftests.
> > 
> > I would tend to agree, yes. Which stand out to you? I've mainly been
> > doing the conversions when I find myself wanting to add new tests, etc.
> lib
> 	test_bitmap
> 	prime_numbers
> 	test_printf
> 	test_scanf

Yeah, these would be nice to convert.

> 	test_strscpy (already converted, need to remove this test)

Yup, converted in bb8d9b742aa7 ("string: Merge strscpy KUnit tests into string_kunit.c")

> lock
> 	test-ww_mutex module
> net
> 	test_blackhole_dev

I don't know these very well, but yeah worth looking into.

> user
> 	test_user_copy (probably already converted, need to remove this test)

This is done in -next via cf6219ee889f ("usercopy: Convert test_user_copy to KUnit test")

> firmware
> 	test_firmware

This might not work to convert: there's a userspace half for testing
firmware loading (see the kselftest side...)

> fpu
> 	test_fpu

Seems reasonable.

> Most of these modules are found in lib/*.
> 
> Would it be desired to move these to kunit?

Checking with the authors/maintainer is probably the first thing to do;
check the git history to see who has been working on them.

-- 
Kees Cook

Re: Converting kselftest test modules to kunit

[John Hubbard]

On 7/16/24 10:59 AM, Kees Cook wrote:
> On Tue, Jul 16, 2024 at 01:11:14PM +0500, Muhammad Usama Anjum wrote:
>> On 7/15/24 9:40 PM, Kees Cook wrote:
>>> On Mon, Jul 15, 2024 at 03:09:24PM +0500, Muhammad Usama Anjum wrote:
>>>> Hi Kees and All,
>>>>
>>>> There are several tests in kselftest subsystem which load modules to tests
>>>> the internals of the kernel. Most of these test modules are just loaded by
>>>> the kselftest, their status isn't read and reported to the user logs. Hence
>>>> they don't provide benefit of executing those tests.
>>>>
>>>> I've found patches from Kees where he has been converting such kselftests
>>>> to kunit tests [1]. The probable motivation is to move tests output of
>>>> kselftest subsystem which only triggers tests without correctly reporting
>>>> the results. On the other hand, kunit is there to test the kernel's
>>>> internal functions which can't be done by userspace.
>>>>
>>>> Kselftest:	Test user facing APIs from userspace
>>>> Kunit:		Test kernel's internal functions from kernelspace
>>>
>>> I would say this is a reasonable guide to how these things should
>>> be separated, yes. That said, much of what was kind of ad-hoc kernel
>>> internals testing that was triggered via kselftests is better done via
>>> KUnit these days, but not everything.
>> I started investigated when I found that kselftest doesn't parse the kernel
>> logs to mark these tests pass/fail. (kselftest/lib is good example of it)
>>
>>>
>>>> This brings me to conclusion that kselftest which are loading modules to
>>>> test kernelspace should be converted to kunit tests. I've noted several
>>>> such kselftests.
>>>
>>> I would tend to agree, yes. Which stand out to you? I've mainly been
>>> doing the conversions when I find myself wanting to add new tests, etc.
>> lib
>> 	test_bitmap
>> 	prime_numbers
>> 	test_printf
>> 	test_scanf
> 
> Yeah, these would be nice to convert.
> 
>> 	test_strscpy (already converted, need to remove this test)
> 
> Yup, converted in bb8d9b742aa7 ("string: Merge strscpy KUnit tests into string_kunit.c")
> 
>> lock
>> 	test-ww_mutex module
>> net
>> 	test_blackhole_dev
> 
> I don't know these very well, but yeah worth looking into.
> 
>> user
>> 	test_user_copy (probably already converted, need to remove this test)
> 
> This is done in -next via cf6219ee889f ("usercopy: Convert test_user_copy to KUnit test")
> 
>> firmware
>> 	test_firmware
> 
> This might not work to convert: there's a userspace half for testing
> firmware loading (see the kselftest side...)
> 
>> fpu
>> 	test_fpu
> 
> Seems reasonable.
> 
>> Most of these modules are found in lib/*.
>>
>> Would it be desired to move these to kunit?
> 
> Checking with the authors/maintainer is probably the first thing to do;
> check the git history to see who has been working on them.
> 

Also maybe:

mm
	gup_test

This is a classic case of testing a kernel API via ioctl into the
kernel, so I think it meets your criteria for moving to KUnit.
Having said that, I must now go try out KUnit in a bit more depth
before I'm sure that this will work out. But it should.

thanks,
-- 
John Hubbard
NVIDIA

Re: Converting kselftest test modules to kunit

[Kees Cook]

On Tue, Jul 16, 2024 at 11:04:06AM -0700, John Hubbard wrote:
> This is a classic case of testing a kernel API via ioctl into the
> kernel, so I think it meets your criteria for moving to KUnit.
> Having said that, I must now go try out KUnit in a bit more depth
> before I'm sure that this will work out. But it should.

I've found the docs really helpful:
https://docs.kernel.org/dev-tools/kunit/

Searching for *_kunit.c will also give a lot of examples. And the kunit
folks are really responsive to questions, helping guide finding solutions
to novel testing needs, etc. :)

-- 
Kees Cook

Re: Converting kselftest test modules to kunit

[Muhammad Usama Anjum]

Hi David,

On 7/16/24 12:33 PM, David Gow wrote:
> On Mon, 15 Jul 2024 at 18:09, Muhammad Usama Anjum
> <usama.anjum@collabora.com> wrote:
>>
>> Hi Kees and All,
>>
>> There are several tests in kselftest subsystem which load modules to tests
>> the internals of the kernel. Most of these test modules are just loaded by
>> the kselftest, their status isn't read and reported to the user logs. Hence
>> they don't provide benefit of executing those tests.
>>
>> I've found patches from Kees where he has been converting such kselftests
>> to kunit tests [1]. The probable motivation is to move tests output of
>> kselftest subsystem which only triggers tests without correctly reporting
>> the results. On the other hand, kunit is there to test the kernel's
>> internal functions which can't be done by userspace.
>>
>> Kselftest:      Test user facing APIs from userspace
>> Kunit:          Test kernel's internal functions from kernelspace
> 
> Yes: this is how we'd like to split things up. There are still a few
> cases where you might want to use kselftest to test something other
> than a user-facing API (if you needed to set up some complicated
> userspace structures, etc), or cases where KUnit might be used to test
> something other than individual pieces of functionality, but that
> categorisation is a good start.
Yeah, makes sense. It is helpful to find out what others think. I'll be
back with changes.

> 
> The Documentation/dev-tools/testing-overview.rst page has a more
> detailed look at when to use which test framework (which basically
> just repeats those rules):
> https://docs.kernel.org/dev-tools/testing-overview.html
> 
> Cheers,
> -- David

-- 
BR,
Muhammad Usama Anjum

Re: Converting kselftest test modules to kunit

[Muhammad Usama Anjum]

On 7/16/24 10:59 PM, Kees Cook wrote:
> On Tue, Jul 16, 2024 at 01:11:14PM +0500, Muhammad Usama Anjum wrote:
>> On 7/15/24 9:40 PM, Kees Cook wrote:
>>> On Mon, Jul 15, 2024 at 03:09:24PM +0500, Muhammad Usama Anjum wrote:
>>>> Hi Kees and All,
>>>>
>>>> There are several tests in kselftest subsystem which load modules to tests
>>>> the internals of the kernel. Most of these test modules are just loaded by
>>>> the kselftest, their status isn't read and reported to the user logs. Hence
>>>> they don't provide benefit of executing those tests.
>>>>
>>>> I've found patches from Kees where he has been converting such kselftests
>>>> to kunit tests [1]. The probable motivation is to move tests output of
>>>> kselftest subsystem which only triggers tests without correctly reporting
>>>> the results. On the other hand, kunit is there to test the kernel's
>>>> internal functions which can't be done by userspace.
>>>>
>>>> Kselftest:	Test user facing APIs from userspace
>>>> Kunit:		Test kernel's internal functions from kernelspace
>>>
>>> I would say this is a reasonable guide to how these things should
>>> be separated, yes. That said, much of what was kind of ad-hoc kernel
>>> internals testing that was triggered via kselftests is better done via
>>> KUnit these days, but not everything.
>> I started investigated when I found that kselftest doesn't parse the kernel
>> logs to mark these tests pass/fail. (kselftest/lib is good example of it)
>>
>>>
>>>> This brings me to conclusion that kselftest which are loading modules to
>>>> test kernelspace should be converted to kunit tests. I've noted several
>>>> such kselftests.
>>>
>>> I would tend to agree, yes. Which stand out to you? I've mainly been
>>> doing the conversions when I find myself wanting to add new tests, etc.
>> lib
>> 	test_bitmap
>> 	prime_numbers
>> 	test_printf
>> 	test_scanf
> 
> Yeah, these would be nice to convert.
I've created TODOs on my end. I'll start working on conversion soon.

> 
>> 	test_strscpy (already converted, need to remove this test)
> 
> Yup, converted in bb8d9b742aa7 ("string: Merge strscpy KUnit tests into string_kunit.c")
> 
>> lock
>> 	test-ww_mutex module
>> net
>> 	test_blackhole_dev
> 
> I don't know these very well, but yeah worth looking into.
> 
>> user
>> 	test_user_copy (probably already converted, need to remove this test)
> 
> This is done in -next via cf6219ee889f ("usercopy: Convert test_user_copy to KUnit test")
> 
>> firmware
>> 	test_firmware
> 
> This might not work to convert: there's a userspace half for testing
> firmware loading (see the kselftest side...)
> 
>> fpu
>> 	test_fpu
> 
> Seems reasonable.
> 
>> Most of these modules are found in lib/*.
>>
>> Would it be desired to move these to kunit?
> 
> Checking with the authors/maintainer is probably the first thing to do;
> check the git history to see who has been working on them.
> 

-- 
BR,
Muhammad Usama Anjum

Re: Converting kselftest test modules to kunit

[John Hubbard]

On 7/16/24 11:26 AM, Kees Cook wrote:
> On Tue, Jul 16, 2024 at 11:04:06AM -0700, John Hubbard wrote:
>> This is a classic case of testing a kernel API via ioctl into the
>> kernel, so I think it meets your criteria for moving to KUnit.
>> Having said that, I must now go try out KUnit in a bit more depth
>> before I'm sure that this will work out. But it should.
> 
> I've found the docs really helpful:
> https://docs.kernel.org/dev-tools/kunit/
> 
> Searching for *_kunit.c will also give a lot of examples. And the kunit
> folks are really responsive to questions, helping guide finding solutions
> to novel testing needs, etc. :)
> 

OK, yes very helpful. I guess I've got some unique biases, because I
found it extremely odd to put so much emphasis on the arch=UM, as
opposed to loading a kunit kernel module on baremetal and running
tests there. I mean, yes, VMs or user mode Linux have their place,
but baremetal testing is really important (and in many cases, the
only way).

Anyway, this confirms for me that kunit should work perfectly for
converting mm/gup_test.

thanks,
-- 
John Hubbard
NVIDIA

Re: Converting kselftest test modules to kunit

[John Hubbard]

On 7/17/24 3:47 AM, Muhammad Usama Anjum wrote:
> Hi David,
> 
> On 7/16/24 12:33 PM, David Gow wrote:
>> On Mon, 15 Jul 2024 at 18:09, Muhammad Usama Anjum
>> <usama.anjum@collabora.com> wrote:
>>>
>>> Hi Kees and All,
>>>
>>> There are several tests in kselftest subsystem which load modules to tests
>>> the internals of the kernel. Most of these test modules are just loaded by
>>> the kselftest, their status isn't read and reported to the user logs. Hence
>>> they don't provide benefit of executing those tests.
>>>
>>> I've found patches from Kees where he has been converting such kselftests
>>> to kunit tests [1]. The probable motivation is to move tests output of
>>> kselftest subsystem which only triggers tests without correctly reporting
>>> the results. On the other hand, kunit is there to test the kernel's
>>> internal functions which can't be done by userspace.
>>>
>>> Kselftest:      Test user facing APIs from userspace
>>> Kunit:          Test kernel's internal functions from kernelspace
>>
>> Yes: this is how we'd like to split things up. There are still a few

Me too. It works.

>> cases where you might want to use kselftest to test something other
>> than a user-facing API (if you needed to set up some complicated
>> userspace structures, etc), or cases where KUnit might be used to test
>> something other than individual pieces of functionality, but that
>> categorisation is a good start.
> Yeah, makes sense. It is helpful to find out what others think. I'll be
> back with changes.
> 

At some point we could connect up the two systems, without really
changing any of the guidelines...much. One way to do that would
be to add a tiny bit of kselftest support for easily launching
a kunit baremetal testing, and then reading the results (which
are right there in sysfs) back to user space.

So it would look like a kselftest, but it could run any kernel-based
tests it needs via kunit.

And the two systems can still be used independently.

>>
>> The Documentation/dev-tools/testing-overview.rst page has a more
>> detailed look at when to use which test framework (which basically
>> just repeats those rules):
>> https://docs.kernel.org/dev-tools/testing-overview.html

Yes, a nice example of up-to-date documentation there. And it's
very clear.

>>
>> Cheers,
>> -- David
> 

thanks,
-- 
John Hubbard
NVIDIA

Re: Converting kselftest test modules to kunit

[Shuah Khan]

On 7/15/24 04:09, Muhammad Usama Anjum wrote:
> Hi Kees and All,
> 
> There are several tests in kselftest subsystem which load modules to tests
> the internals of the kernel. Most of these test modules are just loaded by
> the kselftest, their status isn't read and reported to the user logs. Hence
> they don't provide benefit of executing those tests.
> 
> I've found patches from Kees where he has been converting such kselftests
> to kunit tests [1]. The probable motivation is to move tests output of
> kselftest subsystem which only triggers tests without correctly reporting
> the results. On the other hand, kunit is there to test the kernel's
> internal functions which can't be done by userspace.
> 
> Kselftest:	Test user facing APIs from userspace
> Kunit:		Test kernel's internal functions from kernelspace
> 
> This brings me to conclusion that kselftest which are loading modules to
> test kernelspace should be converted to kunit tests. I've noted several
> such kselftests.
> 
> This is just my understanding. Please mention if I'm correct above or more
> reasons to support kselftest test modules transformation into kunit test.
> 
> [1] https://lore.kernel.org/all/20221018082824.never.845-kees@kernel.org/
> 

Please make sure you aren't taking away the ability to run these tests during
boot. It doesn't make sense to convert every single test especially when it
is intended to be run during boot without dependencies - not as a kunit test
but a regression test during boot.

bitmap is one example - pay attention to the config help test - bitmap
one clearly states it runs regression testing during boot. Any test that
says that isn't a candidate for conversion.

I am going to nack any such conversions.

thanks,
-- Shuah

Re: Converting kselftest test modules to kunit

[Muhammad Usama Anjum]

On 7/27/24 12:35 AM, Shuah Khan wrote:
> On 7/15/24 04:09, Muhammad Usama Anjum wrote:
>> Hi Kees and All,
>>
>> There are several tests in kselftest subsystem which load modules to tests
>> the internals of the kernel. Most of these test modules are just loaded by
>> the kselftest, their status isn't read and reported to the user logs. Hence
>> they don't provide benefit of executing those tests.
>>
>> I've found patches from Kees where he has been converting such kselftests
>> to kunit tests [1]. The probable motivation is to move tests output of
>> kselftest subsystem which only triggers tests without correctly reporting
>> the results. On the other hand, kunit is there to test the kernel's
>> internal functions which can't be done by userspace.
>>
>> Kselftest:    Test user facing APIs from userspace
>> Kunit:        Test kernel's internal functions from kernelspace
>>
>> This brings me to conclusion that kselftest which are loading modules to
>> test kernelspace should be converted to kunit tests. I've noted several
>> such kselftests.
>>
>> This is just my understanding. Please mention if I'm correct above or more
>> reasons to support kselftest test modules transformation into kunit test.
>>
>> [1] https://lore.kernel.org/all/20221018082824.never.845-kees@kernel.org/
>>
> 
> Please make sure you aren't taking away the ability to run these tests during
> boot. 
The kunit tests are usually run at boot time. They can be run later as
well. So I'm not trying to remove any functionality. Rather adding a way
where failures would actually be detected.

> It doesn't make sense to convert every single test especially when it
> is intended to be run during boot without dependencies - not as a kunit test
> but a regression test during boot.
I started investigating when these lib kselftests were just loading the
test module without checking if test actually passed/failed (which proves
that this type of kselftests can never detect regression as it doesn't
process the results). It was strange. Hence I found out the conversion of
such kselftests to kunit tests done by Kees and started this thread for
discussion before doing actual work and sending patches.

> 
> bitmap is one example - pay attention to the config help test - bitmap
> one clearly states it runs regression testing during boot. Any test that
> says that isn't a candidate for conversion.
Please can you explain what do you mean by bitmap testing during boot? As
far as I understand, the kselftests are usespace tests and are run after
kernel has booted completely and userspace has started. It cannot be
regarded as testing during boot.

> 
> I am going to nack any such conversions.
> 
> thanks,
> -- Shuah
> 

-- 
BR,
Muhammad Usama Anjum

Re: Converting kselftest test modules to kunit

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 2763 bytes --]

On Sat, 27 Jul 2024 at 03:35, Shuah Khan <skhan@linuxfoundation.org> wrote:
>
> On 7/15/24 04:09, Muhammad Usama Anjum wrote:
> > Hi Kees and All,
> >
> > There are several tests in kselftest subsystem which load modules to tests
> > the internals of the kernel. Most of these test modules are just loaded by
> > the kselftest, their status isn't read and reported to the user logs. Hence
> > they don't provide benefit of executing those tests.
> >
> > I've found patches from Kees where he has been converting such kselftests
> > to kunit tests [1]. The probable motivation is to move tests output of
> > kselftest subsystem which only triggers tests without correctly reporting
> > the results. On the other hand, kunit is there to test the kernel's
> > internal functions which can't be done by userspace.
> >
> > Kselftest:    Test user facing APIs from userspace
> > Kunit:                Test kernel's internal functions from kernelspace
> >
> > This brings me to conclusion that kselftest which are loading modules to
> > test kernelspace should be converted to kunit tests. I've noted several
> > such kselftests.
> >
> > This is just my understanding. Please mention if I'm correct above or more
> > reasons to support kselftest test modules transformation into kunit test.
> >
> > [1] https://lore.kernel.org/all/20221018082824.never.845-kees@kernel.org/
> >
>
> Please make sure you aren't taking away the ability to run these tests during
> boot. It doesn't make sense to convert every single test especially when it
> is intended to be run during boot without dependencies - not as a kunit test
> but a regression test during boot.

Given KUnit tests can run at boot (and, indeed, do by default if
enabled), I'd've assumed that this would be a good candidate for such
a conversion. It does add the KUnit 'dependency', but I can't think of
how that could be a problem. Is there a specific situation where
enabling CONFIG_KUNIT would cause problems?

> bitmap is one example - pay attention to the config help test - bitmap
> one clearly states it runs regression testing during boot. Any test that
> says that isn't a candidate for conversion.

Again, most KUnit tests are effectively regression tests at boot, so I
don't really understand what makes bitmap different. If it's just a
matter of there being a different interface to it, that's surely
something that we'll either be able to adapt to, or to have some
wrapper/shim to maintain compatibility. I agree that having needless
churn in formats is bad, but KUnit does seem the proper place for
these sorts of tests.

If this isn't the case, do we need to modify the testing guide to
mention this, as it definitely suggests KUnit for tests of in-kernel
functionality like this.

Cheers,
-- David

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4014 bytes --]

Re: Converting kselftest test modules to kunit

[Shuah Khan]

On 7/29/24 23:23, David Gow wrote:
> On Sat, 27 Jul 2024 at 03:35, Shuah Khan <skhan@linuxfoundation.org> wrote:
>>
>> On 7/15/24 04:09, Muhammad Usama Anjum wrote:
>>> Hi Kees and All,
>>>
>>> There are several tests in kselftest subsystem which load modules to tests
>>> the internals of the kernel. Most of these test modules are just loaded by
>>> the kselftest, their status isn't read and reported to the user logs. Hence
>>> they don't provide benefit of executing those tests.
>>>
>>> I've found patches from Kees where he has been converting such kselftests
>>> to kunit tests [1]. The probable motivation is to move tests output of
>>> kselftest subsystem which only triggers tests without correctly reporting
>>> the results. On the other hand, kunit is there to test the kernel's
>>> internal functions which can't be done by userspace.
>>>
>>> Kselftest:    Test user facing APIs from userspace
>>> Kunit:                Test kernel's internal functions from kernelspace
>>>
>>> This brings me to conclusion that kselftest which are loading modules to
>>> test kernelspace should be converted to kunit tests. I've noted several
>>> such kselftests.
>>>
>>> This is just my understanding. Please mention if I'm correct above or more
>>> reasons to support kselftest test modules transformation into kunit test.
>>>
>>> [1] https://lore.kernel.org/all/20221018082824.never.845-kees@kernel.org/
>>>
>>
>> Please make sure you aren't taking away the ability to run these tests during
>> boot. It doesn't make sense to convert every single test especially when it
>> is intended to be run during boot without dependencies - not as a kunit test
>> but a regression test during boot.
> 
> Given KUnit tests can run at boot (and, indeed, do by default if
> enabled), I'd've assumed that this would be a good candidate for such
> a conversion. It does add the KUnit 'dependency', but I can't think of
> how that could be a problem. Is there a specific situation where
> enabling CONFIG_KUNIT would cause problems?
> 
>> bitmap is one example - pay attention to the config help test - bitmap
>> one clearly states it runs regression testing during boot. Any test that
>> says that isn't a candidate for conversion.
> 
> Again, most KUnit tests are effectively regression tests at boot, so I
> don't really understand what makes bitmap different. If it's just a
> matter of there being a different interface to it, that's surely
> something that we'll either be able to adapt to, or to have some
> wrapper/shim to maintain compatibility. I agree that having needless
> churn in formats is bad, but KUnit does seem the proper place for
> these sorts of tests.
> 

The problem is whether not kunit can test at boot time. The issue is
that this type of change removes a selftest which can be run without
kunit dependency thereby removing the ability to run these tests on
a running system without kunit.

This is absolutely necessary for testing on a running system to debug
or sopt check.

> If this isn't the case, do we need to modify the testing guide to
> mention this, as it definitely suggests KUnit for tests of in-kernel
> functionality like this.
> 

Please do. It was never the intent to convert all existing tests toi
kunit.

Repeating what I said on the bitmap removal thread:

This doesn't help people who want run a run bitmap test on a running
system. This is a wrong direction to go to say all testing has to be
done under kunit.

What happened to the effort to run selftests as is under KUnit? What
is the motivation to convert all tests to kunit instead of trying to
provide support to run kselftest under kunit environment?

We discussed this a few years ago as I recall. Let's work on that
instead of removing existing selftests and regressing current use-cases?

Can we look into providing:

1. running kselftest under kunit environment without changes
     as user space applications?

2. Leave kselftests alone so we don't weaken kernel testing by
    removing existing ones and making them dependent on kunit
    thereby removing the existing ability to be able run them on
    non-kunit kernels.


It isn't about kunit vs. kselftest. It is about overall kernel validation
and ability to test effectively by developers and users.

KUnit isn't ideal for cases where people would want to check a subsystem
on a running kernel - KUnit covers some use-cases and kselftest covers
others.

What happens if we are debugging a problem that requires us to debug on
a running system? Please don't go converting kselftest into kunit without
understanding how these frameworks are intended to be used.

Yes kselftest results need to be looked at. Write a parser which can
be improved. What you are doing is reducing the coverage and talking
away the ability to debug and test on running system.

Fix what needs to be fixed instead of deleting tests.

Think through the use-cases before advocating KUnit is suitable for
all testing needs and talking about being able to force or not force
people to use one or the other.

Reports aren't everything. The primary reason we have these tests is for
developers to be able to test. Reports can be improved and shouldn't
come at the expense of coverage and testing. Any patch that does that
will be NACKed.

I already nacked several patches and will continue to nack. Please don't
send me patches that remove existing kselftest tests and turn them into
kunit tests.

thanks,
-- Shuah

Re: Converting kselftest test modules to kunit

[Shuah Khan]

On 7/29/24 01:55, Muhammad Usama Anjum wrote:
> On 7/27/24 12:35 AM, Shuah Khan wrote:
>> On 7/15/24 04:09, Muhammad Usama Anjum wrote:
>>> Hi Kees and All,
>>>
>>> There are several tests in kselftest subsystem which load modules to tests
>>> the internals of the kernel. Most of these test modules are just loaded by
>>> the kselftest, their status isn't read and reported to the user logs. Hence
>>> they don't provide benefit of executing those tests.
>>>
>>> I've found patches from Kees where he has been converting such kselftests
>>> to kunit tests [1]. The probable motivation is to move tests output of
>>> kselftest subsystem which only triggers tests without correctly reporting
>>> the results. On the other hand, kunit is there to test the kernel's
>>> internal functions which can't be done by userspace.
>>>
>>> Kselftest:    Test user facing APIs from userspace
>>> Kunit:        Test kernel's internal functions from kernelspace
>>>
>>> This brings me to conclusion that kselftest which are loading modules to
>>> test kernelspace should be converted to kunit tests. I've noted several
>>> such kselftests.
>>>
>>> This is just my understanding. Please mention if I'm correct above or more
>>> reasons to support kselftest test modules transformation into kunit test.
>>>
>>> [1] https://lore.kernel.org/all/20221018082824.never.845-kees@kernel.org/
>>>
>>
>> Please make sure you aren't taking away the ability to run these tests during
>> boot.
> The kunit tests are usually run at boot time. They can be run later as
> well. So I'm not trying to remove any functionality. Rather adding a way
> where failures would actually be detected.
> 
>> It doesn't make sense to convert every single test especially when it
>> is intended to be run during boot without dependencies - not as a kunit test
>> but a regression test during boot.
> I started investigating when these lib kselftests were just loading the
> test module without checking if test actually passed/failed (which proves
> that this type of kselftests can never detect regression as it doesn't
> process the results). It was strange. Hence I found out the conversion of
> such kselftests to kunit tests done by Kees and started this thread for
> discussion before doing actual work and sending patches.
> 
>>

I explained this on two other threads. If you don't know how this test
is used why are you making this change?

>> bitmap is one example - pay attention to the config help test - bitmap
>> one clearly states it runs regression testing during boot. Any test that
>> says that isn't a candidate for conversion.
> Please can you explain what do you mean by bitmap testing during boot? As
> far as I understand, the kselftests are usespace tests and are run after
> kernel has booted completely and userspace has started. It cannot be
> regarded as testing during boot.
> 

I responded to David - let me know you would like me to repeat it.

thanks,
-- Shuah