[PATCH RFC 0/2] Add support for inlined documentation for kunit and kselftests

[PATCH RFC 0/2] Add support for inlined documentation for kunit and kselftests

[Mauro Carvalho Chehab]

This RFC is a follow-up of the discussions taken here:

   https://lore.kernel.org/linux-doc/20230704132812.02ba97ba@maurocar-mobl2/T/#t

It adds a new extension that allows documenting tests using the same tool we're
using for DRM unit tests at IGT GPU tools: https://gitlab.freedesktop.org/drm/igt-gpu-tools.

While kernel-doc has provided documentation for in-lined functions/struct comments,
it was not meant to document tests.

Tests need to be grouped by the test functions. It should also be possible to produce
other outputs from the documentation, to integrate it with test suites. For instance, 
Internally at Intel, we use the comments to generate DOT files hierarchically grouped
per feature categories.

This is just an initial RFC to start discussions around the solution. Before being merged
upstream, we need to define what tags will be used to identify test markups and add
a simple change at kernel-doc to let it ignore such markups.

On this series, we have:

- patch 1:
  adding test_list.py as present at the IGT tree, after a patch series to make it
  more generic: https://patchwork.freedesktop.org/series/120622/
- patch 2:
  adds an example about how tests could be documented. This is a really simple
  example, just to test the feature, specially designed to make easy to build just
  the test documentation from a single DRM kunit file.

After discussions, my plan is to send a new version addressing the issues, and add
some documentation for DRM and/or i915 kunit tests.

Mauro Carvalho Chehab (2):
  docs: add support for documenting kUnit and kSelftests
  drm: add documentation for drm_buddy_test kUnit test

 Documentation/conf.py                  |    2 +-
 Documentation/index.rst                |    2 +-
 Documentation/sphinx/test_kdoc.py      |  108 ++
 Documentation/sphinx/test_list.py      | 1288 ++++++++++++++++++++++++
 Documentation/tests/index.rst          |    6 +
 Documentation/tests/kunit.rst          |    5 +
 drivers/gpu/drm/tests/drm_buddy_test.c |   12 +
 7 files changed, 1421 insertions(+), 2 deletions(-)
 create mode 100644 Documentation/sphinx/test_kdoc.py
 create mode 100644 Documentation/sphinx/test_list.py
 create mode 100644 Documentation/tests/index.rst
 create mode 100644 Documentation/tests/kunit.rst

-- 
2.40.1

[PATCH RFC 2/2] drm: add documentation for drm_buddy_test kUnit test

[Mauro Carvalho Chehab]

As an example for the new documentation tool, add a documentation
for drm_buddy_test.

I opted to place this on a completely different directory, in order
to make easier to test the feature with:

	$ make SPHINXDIRS="tests" htmldocs

Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---

To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover.
See [PATCH RFC 0/2] at: https://lore.kernel.org/all/cover.1689171160.git.mchehab@kernel.org/

 Documentation/index.rst                |  2 +-
 Documentation/tests/index.rst          |  6 ++++++
 Documentation/tests/kunit.rst          |  5 +++++
 drivers/gpu/drm/tests/drm_buddy_test.c | 12 ++++++++++++
 4 files changed, 24 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/tests/index.rst
 create mode 100644 Documentation/tests/kunit.rst

diff --git a/Documentation/index.rst b/Documentation/index.rst
index 9dfdc826618c..80a6ce14a61a 100644
--- a/Documentation/index.rst
+++ b/Documentation/index.rst
@@ -60,7 +60,7 @@ Various other manuals with useful information for all kernel developers.
    fault-injection/index
    livepatch/index
    rust/index
-
+   test/index
 
 User-oriented documentation
 ===========================
diff --git a/Documentation/tests/index.rst b/Documentation/tests/index.rst
new file mode 100644
index 000000000000..bfc39eb5c0aa
--- /dev/null
+++ b/Documentation/tests/index.rst
@@ -0,0 +1,6 @@
+========================
+Kunit documentation test
+========================
+
+.. toctree::
+   kunit
diff --git a/Documentation/tests/kunit.rst b/Documentation/tests/kunit.rst
new file mode 100644
index 000000000000..6ffc151988a0
--- /dev/null
+++ b/Documentation/tests/kunit.rst
@@ -0,0 +1,5 @@
+Kunit tests
+-----------
+
+.. include-test:: drivers/gpu/drm/tests/drm_buddy_test.c
+
diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c b/drivers/gpu/drm/tests/drm_buddy_test.c
index 09ee6f6af896..dd6c5afd6cd6 100644
--- a/drivers/gpu/drm/tests/drm_buddy_test.c
+++ b/drivers/gpu/drm/tests/drm_buddy_test.c
@@ -737,6 +737,18 @@ static int drm_buddy_suite_init(struct kunit_suite *suite)
 	return 0;
 }
 
+/**
+ * KTEST_SUITE: set of tests for drm buddy alloc
+ * Scope: drm subsystem
+ * Mega feature: drm
+ * Feature: buddy_alloc
+ *
+ * KTEST_TEST: drm_test_buddy_alloc_%s
+ * Description: Run DRM buddy allocation %arg[1] test
+ *
+ * arg[1].values: limit, range, optimistic, smoke, pathological
+ */
+
 static struct kunit_case drm_buddy_tests[] = {
 	KUNIT_CASE(drm_test_buddy_alloc_limit),
 	KUNIT_CASE(drm_test_buddy_alloc_range),
-- 
2.40.1

Re: [PATCH RFC 2/2] drm: add documentation for drm_buddy_test kUnit test

[Jani Nikula]

On Wed, 12 Jul 2023, Mauro Carvalho Chehab <mchehab@kernel.org> wrote:
> diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c b/drivers/gpu/drm/tests/drm_buddy_test.c
> index 09ee6f6af896..dd6c5afd6cd6 100644
> --- a/drivers/gpu/drm/tests/drm_buddy_test.c
> +++ b/drivers/gpu/drm/tests/drm_buddy_test.c
> @@ -737,6 +737,18 @@ static int drm_buddy_suite_init(struct kunit_suite *suite)
>  	return 0;
>  }
>  
> +/**
> + * KTEST_SUITE: set of tests for drm buddy alloc
> + * Scope: drm subsystem
> + * Mega feature: drm
> + * Feature: buddy_alloc
> + *
> + * KTEST_TEST: drm_test_buddy_alloc_%s
> + * Description: Run DRM buddy allocation %arg[1] test
> + *
> + * arg[1].values: limit, range, optimistic, smoke, pathological
> + */
> +

"/**" indicates a kernel-doc comment, and this is not a kernel-doc
comment.

$ scripts/kernel-doc -none drivers/gpu/drm/tests/drm_buddy_test.c 
drivers/gpu/drm/tests/drm_buddy_test.c:752: warning: cannot understand
function prototype: 'struct kunit_case drm_buddy_tests[] = '

Nowadays kernel-doc is part of W=1 builds.


BR,
Jani.


>  static struct kunit_case drm_buddy_tests[] = {
>  	KUNIT_CASE(drm_test_buddy_alloc_limit),
>  	KUNIT_CASE(drm_test_buddy_alloc_range),

-- 
Jani Nikula, Intel Open Source Graphics Center

Re: [PATCH RFC 2/2] drm: add documentation for drm_buddy_test kUnit test

[Mauro Carvalho Chehab]

Em Wed, 12 Jul 2023 18:03:00 +0300
Jani Nikula <jani.nikula@linux.intel.com> escreveu:

> On Wed, 12 Jul 2023, Mauro Carvalho Chehab <mchehab@kernel.org> wrote:
> > diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c b/drivers/gpu/drm/tests/drm_buddy_test.c
> > index 09ee6f6af896..dd6c5afd6cd6 100644
> > --- a/drivers/gpu/drm/tests/drm_buddy_test.c
> > +++ b/drivers/gpu/drm/tests/drm_buddy_test.c
> > @@ -737,6 +737,18 @@ static int drm_buddy_suite_init(struct kunit_suite *suite)
> >  	return 0;
> >  }
> >  
> > +/**
> > + * KTEST_SUITE: set of tests for drm buddy alloc
> > + * Scope: drm subsystem
> > + * Mega feature: drm
> > + * Feature: buddy_alloc
> > + *
> > + * KTEST_TEST: drm_test_buddy_alloc_%s
> > + * Description: Run DRM buddy allocation %arg[1] test
> > + *
> > + * arg[1].values: limit, range, optimistic, smoke, pathological
> > + */
> > +  
> 
> "/**" indicates a kernel-doc comment, and this is not a kernel-doc
> comment.
> 
> $ scripts/kernel-doc -none drivers/gpu/drm/tests/drm_buddy_test.c 
> drivers/gpu/drm/tests/drm_buddy_test.c:752: warning: cannot understand
> function prototype: 'struct kunit_case drm_buddy_tests[] = '
> 
> Nowadays kernel-doc is part of W=1 builds.

True. I already told it at patch 0. I opted to not add a patch for it on this
RFC series, to make it simpler. A simple logic at kernel-doc is enough to 
tell its state machine to ignore blocks that contain the KTEST_\w+: pattern:

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index d0116c6939dc..bf386460691f 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -259,6 +259,7 @@ my $doc_sect = $doc_com .
     '\s*(\@[.\w]+|\@\.\.\.|description|context|returns?|notes?|examples?)\s*:([^:].*)?$';
 my $doc_content = $doc_com_body . '(.*)';
 my $doc_block = $doc_com . 'DOC:\s*(.*)?';
+my $ktest_block = $doc_com . 'KTEST_\w+:\s*(.*)?';
 my $doc_inline_start = '^\s*/\*\*\s*$';
 my $doc_inline_sect = '\s*\*\s*(@\s*[\w][\w\.]*\s*):(.*)';
 my $doc_inline_end = '^\s*\*/\s*$';
@@ -2015,6 +2016,10 @@ sub process_name($$) {
     my $file = shift;
     my $descr;
 
+    if (/$ktest_block/o) {
+	$state = STATE_NORMAL;
+	return;
+    }
     if (/$doc_block/o) {
 	$state = STATE_DOCBLOCK;
 	$contents = "";



Thanks,
Mauro

Re: [PATCH RFC 2/2] drm: add documentation for drm_buddy_test kUnit test

[Christian König]

[Adding Arun]

Am 12.07.23 um 16:28 schrieb Mauro Carvalho Chehab:
> As an example for the new documentation tool, add a documentation
> for drm_buddy_test.
>
> I opted to place this on a completely different directory, in order
> to make easier to test the feature with:
>
> 	$ make SPHINXDIRS="tests" htmldocs
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>

Acked-by: Christian König <christian.koenig@amd.com>

Arun please take a look as well.

Thanks,
Christian.

> ---
>
> To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover.
> See [PATCH RFC 0/2] at: https://lore.kernel.org/all/cover.1689171160.git.mchehab@kernel.org/
>
>   Documentation/index.rst                |  2 +-
>   Documentation/tests/index.rst          |  6 ++++++
>   Documentation/tests/kunit.rst          |  5 +++++
>   drivers/gpu/drm/tests/drm_buddy_test.c | 12 ++++++++++++
>   4 files changed, 24 insertions(+), 1 deletion(-)
>   create mode 100644 Documentation/tests/index.rst
>   create mode 100644 Documentation/tests/kunit.rst
>
> diff --git a/Documentation/index.rst b/Documentation/index.rst
> index 9dfdc826618c..80a6ce14a61a 100644
> --- a/Documentation/index.rst
> +++ b/Documentation/index.rst
> @@ -60,7 +60,7 @@ Various other manuals with useful information for all kernel developers.
>      fault-injection/index
>      livepatch/index
>      rust/index
> -
> +   test/index
>   
>   User-oriented documentation
>   ===========================
> diff --git a/Documentation/tests/index.rst b/Documentation/tests/index.rst
> new file mode 100644
> index 000000000000..bfc39eb5c0aa
> --- /dev/null
> +++ b/Documentation/tests/index.rst
> @@ -0,0 +1,6 @@
> +========================
> +Kunit documentation test
> +========================
> +
> +.. toctree::
> +   kunit
> diff --git a/Documentation/tests/kunit.rst b/Documentation/tests/kunit.rst
> new file mode 100644
> index 000000000000..6ffc151988a0
> --- /dev/null
> +++ b/Documentation/tests/kunit.rst
> @@ -0,0 +1,5 @@
> +Kunit tests
> +-----------
> +
> +.. include-test:: drivers/gpu/drm/tests/drm_buddy_test.c
> +
> diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c b/drivers/gpu/drm/tests/drm_buddy_test.c
> index 09ee6f6af896..dd6c5afd6cd6 100644
> --- a/drivers/gpu/drm/tests/drm_buddy_test.c
> +++ b/drivers/gpu/drm/tests/drm_buddy_test.c
> @@ -737,6 +737,18 @@ static int drm_buddy_suite_init(struct kunit_suite *suite)
>   	return 0;
>   }
>   
> +/**
> + * KTEST_SUITE: set of tests for drm buddy alloc
> + * Scope: drm subsystem
> + * Mega feature: drm
> + * Feature: buddy_alloc
> + *
> + * KTEST_TEST: drm_test_buddy_alloc_%s
> + * Description: Run DRM buddy allocation %arg[1] test
> + *
> + * arg[1].values: limit, range, optimistic, smoke, pathological
> + */
> +
>   static struct kunit_case drm_buddy_tests[] = {
>   	KUNIT_CASE(drm_test_buddy_alloc_limit),
>   	KUNIT_CASE(drm_test_buddy_alloc_range),

Re: [PATCH RFC 2/2] drm: add documentation for drm_buddy_test kUnit test

[Arunpravin Paneer Selvam]

[-- Attachment #1: Type: text/plain, Size: 3305 bytes --]

Looks good, Feel free to add Reviewed-by: Arunpravin Paneer Selvam 
<Arunpravin.PaneerSelvam@amd.com>

Regards,
Arun.

On 7/13/2023 12:09 PM, Christian König wrote:
> [Adding Arun]
>
> Am 12.07.23 um 16:28 schrieb Mauro Carvalho Chehab:
>> As an example for the new documentation tool, add a documentation
>> for drm_buddy_test.
>>
>> I opted to place this on a completely different directory, in order
>> to make easier to test the feature with:
>>
>>     $ make SPHINXDIRS="tests" htmldocs
>>
>> Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
>
> Acked-by: Christian König <christian.koenig@amd.com>
>
> Arun please take a look as well.
>
> Thanks,
> Christian.
>
>> ---
>>
>> To avoid mailbombing on a large number of people, only mailing lists 
>> were C/C on the cover.
>> See [PATCH RFC 0/2] at: 
>> https://lore.kernel.org/all/cover.1689171160.git.mchehab@kernel.org/
>>
>>   Documentation/index.rst                |  2 +-
>>   Documentation/tests/index.rst          |  6 ++++++
>>   Documentation/tests/kunit.rst          |  5 +++++
>>   drivers/gpu/drm/tests/drm_buddy_test.c | 12 ++++++++++++
>>   4 files changed, 24 insertions(+), 1 deletion(-)
>>   create mode 100644 Documentation/tests/index.rst
>>   create mode 100644 Documentation/tests/kunit.rst
>>
>> diff --git a/Documentation/index.rst b/Documentation/index.rst
>> index 9dfdc826618c..80a6ce14a61a 100644
>> --- a/Documentation/index.rst
>> +++ b/Documentation/index.rst
>> @@ -60,7 +60,7 @@ Various other manuals with useful information for 
>> all kernel developers.
>>      fault-injection/index
>>      livepatch/index
>>      rust/index
>> -
>> +   test/index
>>     User-oriented documentation
>>   ===========================
>> diff --git a/Documentation/tests/index.rst 
>> b/Documentation/tests/index.rst
>> new file mode 100644
>> index 000000000000..bfc39eb5c0aa
>> --- /dev/null
>> +++ b/Documentation/tests/index.rst
>> @@ -0,0 +1,6 @@
>> +========================
>> +Kunit documentation test
>> +========================
>> +
>> +.. toctree::
>> +   kunit
>> diff --git a/Documentation/tests/kunit.rst 
>> b/Documentation/tests/kunit.rst
>> new file mode 100644
>> index 000000000000..6ffc151988a0
>> --- /dev/null
>> +++ b/Documentation/tests/kunit.rst
>> @@ -0,0 +1,5 @@
>> +Kunit tests
>> +-----------
>> +
>> +.. include-test:: drivers/gpu/drm/tests/drm_buddy_test.c
>> +
>> diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c 
>> b/drivers/gpu/drm/tests/drm_buddy_test.c
>> index 09ee6f6af896..dd6c5afd6cd6 100644
>> --- a/drivers/gpu/drm/tests/drm_buddy_test.c
>> +++ b/drivers/gpu/drm/tests/drm_buddy_test.c
>> @@ -737,6 +737,18 @@ static int drm_buddy_suite_init(struct 
>> kunit_suite *suite)
>>       return 0;
>>   }
>>   +/**
>> + * KTEST_SUITE: set of tests for drm buddy alloc
>> + * Scope: drm subsystem
>> + * Mega feature: drm
>> + * Feature: buddy_alloc
>> + *
>> + * KTEST_TEST: drm_test_buddy_alloc_%s
>> + * Description: Run DRM buddy allocation %arg[1] test
>> + *
>> + * arg[1].values: limit, range, optimistic, smoke, pathological
>> + */
>> +
>>   static struct kunit_case drm_buddy_tests[] = {
>>       KUNIT_CASE(drm_test_buddy_alloc_limit),
>>       KUNIT_CASE(drm_test_buddy_alloc_range),
>

[-- Attachment #2: Type: text/html, Size: 5983 bytes --]

Re: [PATCH RFC 2/2] drm: add documentation for drm_buddy_test kUnit test

[Rae Moar]

[-- Attachment #1: Type: text/plain, Size: 4180 bytes --]

On Wed, Jul 12, 2023 at 10:29 AM Mauro Carvalho Chehab <mchehab@kernel.org>
wrote:

> As an example for the new documentation tool, add a documentation
> for drm_buddy_test.
>
> I opted to place this on a completely different directory, in order
> to make easier to test the feature with:
>
>         $ make SPHINXDIRS="tests" htmldocs
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
> ---
>
> To avoid mailbombing on a large number of people, only mailing lists were
> C/C on the cover.
> See [PATCH RFC 0/2] at:
> https://lore.kernel.org/all/cover.1689171160.git.mchehab@kernel.org/
>
>  Documentation/index.rst                |  2 +-
>  Documentation/tests/index.rst          |  6 ++++++
>  Documentation/tests/kunit.rst          |  5 +++++
>  drivers/gpu/drm/tests/drm_buddy_test.c | 12 ++++++++++++
>  4 files changed, 24 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/tests/index.rst
>  create mode 100644 Documentation/tests/kunit.rst
>
> diff --git a/Documentation/index.rst b/Documentation/index.rst
> index 9dfdc826618c..80a6ce14a61a 100644
> --- a/Documentation/index.rst
> +++ b/Documentation/index.rst
> @@ -60,7 +60,7 @@ Various other manuals with useful information for all
> kernel developers.
>     fault-injection/index
>     livepatch/index
>     rust/index
> -
> +   test/index
>
>  User-oriented documentation
>  ===========================
> diff --git a/Documentation/tests/index.rst b/Documentation/tests/index.rst
> new file mode 100644
> index 000000000000..bfc39eb5c0aa
> --- /dev/null
> +++ b/Documentation/tests/index.rst
> @@ -0,0 +1,6 @@
> +========================
> +Kunit documentation test
> +========================
> +
> +.. toctree::
> +   kunit
> diff --git a/Documentation/tests/kunit.rst b/Documentation/tests/kunit.rst
> new file mode 100644
> index 000000000000..6ffc151988a0
> --- /dev/null
> +++ b/Documentation/tests/kunit.rst
> @@ -0,0 +1,5 @@
> +Kunit tests
> +-----------
> +
> +.. include-test:: drivers/gpu/drm/tests/drm_buddy_test.c
> +
> diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c
> b/drivers/gpu/drm/tests/drm_buddy_test.c
> index 09ee6f6af896..dd6c5afd6cd6 100644
> --- a/drivers/gpu/drm/tests/drm_buddy_test.c
> +++ b/drivers/gpu/drm/tests/drm_buddy_test.c
> @@ -737,6 +737,18 @@ static int drm_buddy_suite_init(struct kunit_suite
> *suite)
>         return 0;
>  }
>
> +/**
> + * KTEST_SUITE: set of tests for drm buddy alloc
> + * Scope: drm subsystem
> + * Mega feature: drm
> + * Feature: buddy_alloc
> + *
> + * KTEST_TEST: drm_test_buddy_alloc_%s
> + * Description: Run DRM buddy allocation %arg[1] test
> + *
> + * arg[1].values: limit, range, optimistic, smoke, pathological
> + */


Hi!

This is such a cool patch series. I just have a few comments related to the
output.

In the html output the tests are listed as:
ktest@drm_buddy_test@…

I wonder if instead of using the file name of “drm_buddy_test” this could
possibly be the suite name, “drm_buddy”, as this is what users will call
when using kunit.py to run the tests. Although "drm_buddy_test" is also the
module name so I don't mind it too much. But in the future the file name
and module name are not guaranteed to be the same for other tests.

Most preferably, there would be a reference to the kunit suite name, file
name, and the module name.

This may be difficult to implement as these can all differ. I am currently
working on the KUnit Attribute framework which saves the module name and I
am thinking about also saving the file path as a future attribute. This
could be a helpful framework for the KUnit tests specifically.

I am not sure how easy it would be to access c objects/functions using this
system.

Finally, I was wondering if it is the intention to put a list of all kunit
tests that use this new feature into tests/kunit.rst or would this be
broken up in some way

Thanks!
-Rae



> +
>  static struct kunit_case drm_buddy_tests[] = {
>         KUNIT_CASE(drm_test_buddy_alloc_limit),
>         KUNIT_CASE(drm_test_buddy_alloc_range),
> --
> 2.40.1
>
>

[-- Attachment #2: Type: text/html, Size: 5445 bytes --]

Re: [PATCH RFC 2/2] drm: add documentation for drm_buddy_test kUnit test

[Rae Moar]

On Wed, Jul 12, 2023 at 10:29 AM Mauro Carvalho Chehab
<mchehab@kernel.org> wrote:
>
> As an example for the new documentation tool, add a documentation
> for drm_buddy_test.
>
> I opted to place this on a completely different directory, in order
> to make easier to test the feature with:
>
>         $ make SPHINXDIRS="tests" htmldocs
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
> ---
>
> To avoid mailbombing on a large number of people, only mailing lists were C/C on the cover.
> See [PATCH RFC 0/2] at: https://lore.kernel.org/all/cover.1689171160.git.mchehab@kernel.org/
>
>  Documentation/index.rst                |  2 +-
>  Documentation/tests/index.rst          |  6 ++++++
>  Documentation/tests/kunit.rst          |  5 +++++
>  drivers/gpu/drm/tests/drm_buddy_test.c | 12 ++++++++++++
>  4 files changed, 24 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/tests/index.rst
>  create mode 100644 Documentation/tests/kunit.rst
>
> diff --git a/Documentation/index.rst b/Documentation/index.rst
> index 9dfdc826618c..80a6ce14a61a 100644
> --- a/Documentation/index.rst
> +++ b/Documentation/index.rst
> @@ -60,7 +60,7 @@ Various other manuals with useful information for all kernel developers.
>     fault-injection/index
>     livepatch/index
>     rust/index
> -
> +   test/index
>
>  User-oriented documentation
>  ===========================
> diff --git a/Documentation/tests/index.rst b/Documentation/tests/index.rst
> new file mode 100644
> index 000000000000..bfc39eb5c0aa
> --- /dev/null
> +++ b/Documentation/tests/index.rst
> @@ -0,0 +1,6 @@
> +========================
> +Kunit documentation test
> +========================
> +
> +.. toctree::
> +   kunit
> diff --git a/Documentation/tests/kunit.rst b/Documentation/tests/kunit.rst
> new file mode 100644
> index 000000000000..6ffc151988a0
> --- /dev/null
> +++ b/Documentation/tests/kunit.rst
> @@ -0,0 +1,5 @@
> +Kunit tests
> +-----------
> +
> +.. include-test:: drivers/gpu/drm/tests/drm_buddy_test.c
> +
> diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c b/drivers/gpu/drm/tests/drm_buddy_test.c
> index 09ee6f6af896..dd6c5afd6cd6 100644
> --- a/drivers/gpu/drm/tests/drm_buddy_test.c
> +++ b/drivers/gpu/drm/tests/drm_buddy_test.c
> @@ -737,6 +737,18 @@ static int drm_buddy_suite_init(struct kunit_suite *suite)
>         return 0;
>  }
>
> +/**
> + * KTEST_SUITE: set of tests for drm buddy alloc
> + * Scope: drm subsystem
> + * Mega feature: drm
> + * Feature: buddy_alloc
> + *
> + * KTEST_TEST: drm_test_buddy_alloc_%s
> + * Description: Run DRM buddy allocation %arg[1] test
> + *
> + * arg[1].values: limit, range, optimistic, smoke, pathological
> + */
> +

I apologize that the last email included a HTML attachment of the message.

Just in case anyone was unable to receive the last email here is a
copy of the message:

Hi!

This is such a cool patch series. I just have a few comments related
to the output.

In the html output the tests are listed as:
ktest@drm_buddy_test@...

I wonder if instead of using the file name of "drm_buddy_test" this
could possibly be the suite name, "drm_buddy", as this is what users
will call when using kunit.py to run the tests. Although
"drm_buddy_test" is also the module name so I don't mind it too much.
But in the future the file name and module name are not guaranteed to
be the same for other tests.

Most preferably, there would be a reference to the kunit suite name,
file name, and the module name.

This may be difficult to implement as these can all differ. I am
currently working on the KUnit Attribute framework which saves the
module name and I am thinking about also saving the file path as a
future attribute. This could be a helpful framework for the KUnit
tests specifically.

I am not sure how easy it would be to access c objects/functions using
this system.

Finally, I was wondering if it is the intention to put a list of all
KUnit tests that use this new feature into tests/kunit.rst or would
this be broken up in some way.

Thanks!
-Rae

>  static struct kunit_case drm_buddy_tests[] = {
>         KUNIT_CASE(drm_test_buddy_alloc_limit),
>         KUNIT_CASE(drm_test_buddy_alloc_range),
> --
> 2.40.1
>

Re: [PATCH RFC 2/2] drm: add documentation for drm_buddy_test kUnit test

[Mauro Carvalho Chehab]

Hi Rae,

Em Thu, 13 Jul 2023 17:31:19 -0400
Rae Moar <rmoar@google.com> escreveu:

> On Wed, Jul 12, 2023 at 10:29 AM Mauro Carvalho Chehab <mchehab@kernel.org>
> wrote:
> 
> > As an example for the new documentation tool, add a documentation
> > for drm_buddy_test.
> >
> > I opted to place this on a completely different directory, in order
> > to make easier to test the feature with:
> >
> >         $ make SPHINXDIRS="tests" htmldocs
> >
> > Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
> > ---
> >
> > To avoid mailbombing on a large number of people, only mailing lists were
> > C/C on the cover.
> > See [PATCH RFC 0/2] at:
> > https://lore.kernel.org/all/cover.1689171160.git.mchehab@kernel.org/
> >
> >  Documentation/index.rst                |  2 +-
> >  Documentation/tests/index.rst          |  6 ++++++
> >  Documentation/tests/kunit.rst          |  5 +++++
> >  drivers/gpu/drm/tests/drm_buddy_test.c | 12 ++++++++++++
> >  4 files changed, 24 insertions(+), 1 deletion(-)
> >  create mode 100644 Documentation/tests/index.rst
> >  create mode 100644 Documentation/tests/kunit.rst
> >
> > diff --git a/Documentation/index.rst b/Documentation/index.rst
> > index 9dfdc826618c..80a6ce14a61a 100644
> > --- a/Documentation/index.rst
> > +++ b/Documentation/index.rst
> > @@ -60,7 +60,7 @@ Various other manuals with useful information for all
> > kernel developers.
> >     fault-injection/index
> >     livepatch/index
> >     rust/index
> > -
> > +   test/index
> >
> >  User-oriented documentation
> >  ===========================
> > diff --git a/Documentation/tests/index.rst b/Documentation/tests/index.rst
> > new file mode 100644
> > index 000000000000..bfc39eb5c0aa
> > --- /dev/null
> > +++ b/Documentation/tests/index.rst
> > @@ -0,0 +1,6 @@
> > +========================
> > +Kunit documentation test
> > +========================
> > +
> > +.. toctree::
> > +   kunit
> > diff --git a/Documentation/tests/kunit.rst b/Documentation/tests/kunit.rst
> > new file mode 100644
> > index 000000000000..6ffc151988a0
> > --- /dev/null
> > +++ b/Documentation/tests/kunit.rst
> > @@ -0,0 +1,5 @@
> > +Kunit tests
> > +-----------
> > +
> > +.. include-test:: drivers/gpu/drm/tests/drm_buddy_test.c
> > +
> > diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c
> > b/drivers/gpu/drm/tests/drm_buddy_test.c
> > index 09ee6f6af896..dd6c5afd6cd6 100644
> > --- a/drivers/gpu/drm/tests/drm_buddy_test.c
> > +++ b/drivers/gpu/drm/tests/drm_buddy_test.c
> > @@ -737,6 +737,18 @@ static int drm_buddy_suite_init(struct kunit_suite
> > *suite)
> >         return 0;
> >  }
> >
> > +/**
> > + * KTEST_SUITE: set of tests for drm buddy alloc
> > + * Scope: drm subsystem
> > + * Mega feature: drm
> > + * Feature: buddy_alloc
> > + *
> > + * KTEST_TEST: drm_test_buddy_alloc_%s
> > + * Description: Run DRM buddy allocation %arg[1] test
> > + *
> > + * arg[1].values: limit, range, optimistic, smoke, pathological
> > + */  
> 
> 
> Hi!
> 
> This is such a cool patch series. I just have a few comments related to the
> output.

Thank you for your comments! Sorry for not answering earlier. I took some
vacations and this series ended sleeping over other tasks on my
todo list.

Also, before sending another version, I wanted to have the test_list.py
changes to make it generic enough to be merged on IGT, to avoid having
a fork of it. Those got merged today.

> In the html output the tests are listed as:
> ktest@drm_buddy_test@…
> 
> I wonder if instead of using the file name of “drm_buddy_test” this could
> possibly be the suite name, “drm_buddy”, as this is what users will call
> when using kunit.py to run the tests. Although "drm_buddy_test" is also the
> module name so I don't mind it too much. But in the future the file name
> and module name are not guaranteed to be the same for other tests.
> 
> Most preferably, there would be a reference to the kunit suite name, file
> name, and the module name.

I guess it shouldn't be hard to do such change in a way that it won't
affect its usage on IGT. We need to define what would be the best
pattern. As this can be used for both kunit and selftests, I would
place kunit at the beginning.

Currently, we're using:

	kunit@<base file name without .c>@<test_name>

Some possible patterns would be:

	kunit@<base file name without .c>@<suite name>@<test_name>
	kunit@<subsystem>@<base file name without .c>@<suite name>@<test_name>
	kunit@<subsystem>@<suite name>@<test_name>

Would do you think it would work best?

> This may be difficult to implement as these can all differ. I am currently
> working on the KUnit Attribute framework which saves the module name and I
> am thinking about also saving the file path as a future attribute. This
> could be a helpful framework for the KUnit tests specifically.
> 
> I am not sure how easy it would be to access c objects/functions using this
> system.

I would prefer not. C language allows lots of flexibility with macros,
making it hard to write a parser to read those C objects from the source.
We have this at kernel-doc. As one of the people that did some work there,
I can say that that several tricks are needed to keep this working.

On the other hand, it should be easy to use the TestList class from
test_list.py at kunit.py.

So, kunit.py could use the data that came from the documentation
directly.

> Finally, I was wondering if it is the intention to put a list of all kunit
> tests that use this new feature into tests/kunit.rst or would this be
> broken up in some way

IMO, it makes sense to break this per subsystem, and have an auto-generated
index.rst pointing to the entire set of documents.

We're already storing the subsystem at the documentation macros, so, IMO,
it should shouldn't be hard to implement it.

Regards,
Mauro

Re: [PATCH RFC 2/2] drm: add documentation for drm_buddy_test kUnit test

[Rae Moar]

On Fri, Sep 1, 2023 at 3:11 AM Mauro Carvalho Chehab <mchehab@kernel.org> wrote:
>
> Hi Rae,
>
> Em Thu, 13 Jul 2023 17:31:19 -0400
> Rae Moar <rmoar@google.com> escreveu:
>
> > On Wed, Jul 12, 2023 at 10:29 AM Mauro Carvalho Chehab <mchehab@kernel.org>
> > wrote:
> >
> > > As an example for the new documentation tool, add a documentation
> > > for drm_buddy_test.
> > >
> > > I opted to place this on a completely different directory, in order
> > > to make easier to test the feature with:
> > >
> > >         $ make SPHINXDIRS="tests" htmldocs
> > >
> > > Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
> > > ---
> > >
> > > To avoid mailbombing on a large number of people, only mailing lists were
> > > C/C on the cover.
> > > See [PATCH RFC 0/2] at:
> > > https://lore.kernel.org/all/cover.1689171160.git.mchehab@kernel.org/
> > >
> > >  Documentation/index.rst                |  2 +-
> > >  Documentation/tests/index.rst          |  6 ++++++
> > >  Documentation/tests/kunit.rst          |  5 +++++
> > >  drivers/gpu/drm/tests/drm_buddy_test.c | 12 ++++++++++++
> > >  4 files changed, 24 insertions(+), 1 deletion(-)
> > >  create mode 100644 Documentation/tests/index.rst
> > >  create mode 100644 Documentation/tests/kunit.rst
> > >
> > > diff --git a/Documentation/index.rst b/Documentation/index.rst
> > > index 9dfdc826618c..80a6ce14a61a 100644
> > > --- a/Documentation/index.rst
> > > +++ b/Documentation/index.rst
> > > @@ -60,7 +60,7 @@ Various other manuals with useful information for all
> > > kernel developers.
> > >     fault-injection/index
> > >     livepatch/index
> > >     rust/index
> > > -
> > > +   test/index
> > >
> > >  User-oriented documentation
> > >  ===========================
> > > diff --git a/Documentation/tests/index.rst b/Documentation/tests/index.rst
> > > new file mode 100644
> > > index 000000000000..bfc39eb5c0aa
> > > --- /dev/null
> > > +++ b/Documentation/tests/index.rst
> > > @@ -0,0 +1,6 @@
> > > +========================
> > > +Kunit documentation test
> > > +========================
> > > +
> > > +.. toctree::
> > > +   kunit
> > > diff --git a/Documentation/tests/kunit.rst b/Documentation/tests/kunit.rst
> > > new file mode 100644
> > > index 000000000000..6ffc151988a0
> > > --- /dev/null
> > > +++ b/Documentation/tests/kunit.rst
> > > @@ -0,0 +1,5 @@
> > > +Kunit tests
> > > +-----------
> > > +
> > > +.. include-test:: drivers/gpu/drm/tests/drm_buddy_test.c
> > > +
> > > diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c
> > > b/drivers/gpu/drm/tests/drm_buddy_test.c
> > > index 09ee6f6af896..dd6c5afd6cd6 100644
> > > --- a/drivers/gpu/drm/tests/drm_buddy_test.c
> > > +++ b/drivers/gpu/drm/tests/drm_buddy_test.c
> > > @@ -737,6 +737,18 @@ static int drm_buddy_suite_init(struct kunit_suite
> > > *suite)
> > >         return 0;
> > >  }
> > >
> > > +/**
> > > + * KTEST_SUITE: set of tests for drm buddy alloc
> > > + * Scope: drm subsystem
> > > + * Mega feature: drm
> > > + * Feature: buddy_alloc
> > > + *
> > > + * KTEST_TEST: drm_test_buddy_alloc_%s
> > > + * Description: Run DRM buddy allocation %arg[1] test
> > > + *
> > > + * arg[1].values: limit, range, optimistic, smoke, pathological
> > > + */
> >
> >
> > Hi!
> >
> > This is such a cool patch series. I just have a few comments related to the
> > output.
>
> Thank you for your comments! Sorry for not answering earlier. I took some
> vacations and this series ended sleeping over other tasks on my
> todo list.
>
> Also, before sending another version, I wanted to have the test_list.py
> changes to make it generic enough to be merged on IGT, to avoid having
> a fork of it. Those got merged today.

Hi Mauro!

No worries at all!

>
> > In the html output the tests are listed as:
> > ktest@drm_buddy_test@…
> >
> > I wonder if instead of using the file name of “drm_buddy_test” this could
> > possibly be the suite name, “drm_buddy”, as this is what users will call
> > when using kunit.py to run the tests. Although "drm_buddy_test" is also the
> > module name so I don't mind it too much. But in the future the file name
> > and module name are not guaranteed to be the same for other tests.
> >
> > Most preferably, there would be a reference to the kunit suite name, file
> > name, and the module name.
>
> I guess it shouldn't be hard to do such change in a way that it won't
> affect its usage on IGT. We need to define what would be the best
> pattern. As this can be used for both kunit and selftests, I would
> place kunit at the beginning.
>
> Currently, we're using:
>
>         kunit@<base file name without .c>@<test_name>
>
> Some possible patterns would be:
>
>         kunit@<base file name without .c>@<suite name>@<test_name>
>         kunit@<subsystem>@<base file name without .c>@<suite name>@<test_name>
>         kunit@<subsystem>@<suite name>@<test_name>
>
> Would do you think it would work best?

How possible is it to separate out the file and suite name as headers?
I think that this could reduce some of the repetition.

If we are already separating documentation pages by subsystem, a
potential format could be:

File: <base file>

<kunit_or_kselftest> suite: <suite name>
List of Tests:
<test name>
<test name>
...

What do you think?

>
> > This may be difficult to implement as these can all differ. I am currently
> > working on the KUnit Attribute framework which saves the module name and I
> > am thinking about also saving the file path as a future attribute. This
> > could be a helpful framework for the KUnit tests specifically.
> >
> > I am not sure how easy it would be to access c objects/functions using this
> > system.
>
> I would prefer not. C language allows lots of flexibility with macros,
> making it hard to write a parser to read those C objects from the source.
> We have this at kernel-doc. As one of the people that did some work there,
> I can say that that several tricks are needed to keep this working.
>
> On the other hand, it should be easy to use the TestList class from
> test_list.py at kunit.py.
>
> So, kunit.py could use the data that came from the documentation
> directly.
>

Got it. So it is possible to get some of this info. Thanks!

> > Finally, I was wondering if it is the intention to put a list of all kunit
> > tests that use this new feature into tests/kunit.rst or would this be
> > broken up in some way
>
> IMO, it makes sense to break this per subsystem, and have an auto-generated
> index.rst pointing to the entire set of documents.
>
> We're already storing the subsystem at the documentation macros, so, IMO,
> it should shouldn't be hard to implement it.
>
> Regards,
> Mauro

I think breaking this up by subsystems sounds like a good idea,
especially since we still have them documented already.

Thanks for your responses!
-Rae

>