[PATCH 0/2] Add support for inlined documentation for kunit and kselftests

[PATCH 0/2] Add support for inlined documentation for kunit and kselftests

[Mauro Carvalho Chehab]

This is a follow-up of the discussions taken here:

   https://lore.kernel.org/linux-doc/20230704132812.02ba97ba@maurocar-mobl2/T/#t

I sent a previous version as RFC. This is basically what we had there, with some
improvements at test_list.py.

It adds a new extension that allows documenting tests using the same tool we're
using for DRM unit tests at IGT GPU tools: https://gitlab.freedesktop.org/drm/igt-gpu-tools.

While kernel-doc has provided documentation for in-lined functions/struct comments,
it was not meant to document tests.

Tests need to be grouped by the test functions. It should also be possible to produce
other outputs from the documentation, to integrate it with test suites. For instance, 
Internally at Intel, we use the comments to generate DOT files hierarchically grouped
per feature categories.

This is meant to be an initial series to start documenting kunit.

It comes with a documenation for a kunit suite from drm at drm_buddy_test.c.

My plan is that, once we get this merged, start documenting other unit tests and work
to add extra requirements at the tool needed by Linux Kernel.

On this series, we have:

- patch 1:
  adds test_list.py - which is identical to the file with the same name I developed for
  IGT tree;
- patch 2:
  adds  test documentation from a single DRM kunit file (drm_buddy_test.c).

Mauro Carvalho Chehab (2):
  docs: add support for documenting kUnit and kSelftests
  drm: add documentation for drm_buddy_test kUnit test

 Documentation/conf.py                  |    2 +-
 Documentation/index.rst                |    2 +-
 Documentation/sphinx/test_kdoc.py      |  108 ++
 Documentation/sphinx/test_list.py      | 1314 ++++++++++++++++++++++++
 Documentation/tests/index.rst          |    6 +
 Documentation/tests/kunit.rst          |    5 +
 drivers/gpu/drm/tests/drm_buddy_test.c |   12 +
 7 files changed, 1447 insertions(+), 2 deletions(-)
 create mode 100755 Documentation/sphinx/test_kdoc.py
 create mode 100644 Documentation/sphinx/test_list.py
 create mode 100644 Documentation/tests/index.rst
 create mode 100644 Documentation/tests/kunit.rst

-- 
2.41.0

[PATCH 2/2] drm: add documentation for drm_buddy_test kUnit test

[Mauro Carvalho Chehab]

As an example for the new documentation tool, add a documentation
for drm_buddy_test.

I opted to place this on a completely different directory, in order
to make easier to test the feature with:

	$ make SPHINXDIRS="tests" htmldocs

Acked-by: Christian König <christian.koening@amd.com>
Reviewed-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
---
 Documentation/index.rst                |  2 +-
 Documentation/tests/index.rst          |  6 ++++++
 Documentation/tests/kunit.rst          |  5 +++++
 drivers/gpu/drm/tests/drm_buddy_test.c | 12 ++++++++++++
 4 files changed, 24 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/tests/index.rst
 create mode 100644 Documentation/tests/kunit.rst

diff --git a/Documentation/index.rst b/Documentation/index.rst
index 9dfdc826618c..80a6ce14a61a 100644
--- a/Documentation/index.rst
+++ b/Documentation/index.rst
@@ -60,7 +60,7 @@ Various other manuals with useful information for all kernel developers.
    fault-injection/index
    livepatch/index
    rust/index
-
+   test/index
 
 User-oriented documentation
 ===========================
diff --git a/Documentation/tests/index.rst b/Documentation/tests/index.rst
new file mode 100644
index 000000000000..bfc39eb5c0aa
--- /dev/null
+++ b/Documentation/tests/index.rst
@@ -0,0 +1,6 @@
+========================
+Kunit documentation test
+========================
+
+.. toctree::
+   kunit
diff --git a/Documentation/tests/kunit.rst b/Documentation/tests/kunit.rst
new file mode 100644
index 000000000000..6ffc151988a0
--- /dev/null
+++ b/Documentation/tests/kunit.rst
@@ -0,0 +1,5 @@
+Kunit tests
+-----------
+
+.. include-test:: drivers/gpu/drm/tests/drm_buddy_test.c
+
diff --git a/drivers/gpu/drm/tests/drm_buddy_test.c b/drivers/gpu/drm/tests/drm_buddy_test.c
index 09ee6f6af896..dd6c5afd6cd6 100644
--- a/drivers/gpu/drm/tests/drm_buddy_test.c
+++ b/drivers/gpu/drm/tests/drm_buddy_test.c
@@ -737,6 +737,18 @@ static int drm_buddy_suite_init(struct kunit_suite *suite)
 	return 0;
 }
 
+/**
+ * KTEST_SUITE: set of tests for drm buddy alloc
+ * Scope: drm subsystem
+ * Mega feature: drm
+ * Feature: buddy_alloc
+ *
+ * KTEST_TEST: drm_test_buddy_alloc_%s
+ * Description: Run DRM buddy allocation %arg[1] test
+ *
+ * arg[1].values: limit, range, optimistic, smoke, pathological
+ */
+
 static struct kunit_case drm_buddy_tests[] = {
 	KUNIT_CASE(drm_test_buddy_alloc_limit),
 	KUNIT_CASE(drm_test_buddy_alloc_range),
-- 
2.41.0

Re: [PATCH 0/2] Add support for inlined documentation for kunit and kselftests

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab@kernel.org> writes:

> This is a follow-up of the discussions taken here:
>
>    https://lore.kernel.org/linux-doc/20230704132812.02ba97ba@maurocar-mobl2/T/#t
>
> I sent a previous version as RFC. This is basically what we had there, with some
> improvements at test_list.py.
>
> It adds a new extension that allows documenting tests using the same tool we're
> using for DRM unit tests at IGT GPU tools: https://gitlab.freedesktop.org/drm/igt-gpu-tools.
>
> While kernel-doc has provided documentation for in-lined functions/struct comments,
> it was not meant to document tests.
>
> Tests need to be grouped by the test functions. It should also be possible to produce
> other outputs from the documentation, to integrate it with test suites. For instance, 
> Internally at Intel, we use the comments to generate DOT files hierarchically grouped
> per feature categories.
>
> This is meant to be an initial series to start documenting kunit.

I've played with this a bit...a couple of quick impressions:

- That's quite a chunk of Python code to be adding.  I've not yet had
  the chance to read it through properly, will hopefully be able to do
  so soon.  A bit more commenting would not have gone amiss here...

- I kind of think that this should go under dev-tools rather than being
  a new top-level directory.  Is there a reason not to put it there?

Thanks,

jon

Re: [PATCH 2/2] drm: add documentation for drm_buddy_test kUnit test

[Jonathan Corbet]

One other little thing...

Mauro Carvalho Chehab <mchehab@kernel.org> writes:

> As an example for the new documentation tool, add a documentation
> for drm_buddy_test.
>
> I opted to place this on a completely different directory, in order
> to make easier to test the feature with:
>
> 	$ make SPHINXDIRS="tests" htmldocs
>
> Acked-by: Christian König <christian.koening@amd.com>
> Reviewed-by: Arunpravin Paneer Selvam <Arunpravin.PaneerSelvam@amd.com>
> Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
> ---
>  Documentation/index.rst                |  2 +-
>  Documentation/tests/index.rst          |  6 ++++++
>  Documentation/tests/kunit.rst          |  5 +++++
>  drivers/gpu/drm/tests/drm_buddy_test.c | 12 ++++++++++++
>  4 files changed, 24 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/tests/index.rst
>  create mode 100644 Documentation/tests/kunit.rst
>
> diff --git a/Documentation/index.rst b/Documentation/index.rst
> index 9dfdc826618c..80a6ce14a61a 100644
> --- a/Documentation/index.rst
> +++ b/Documentation/index.rst
> @@ -60,7 +60,7 @@ Various other manuals with useful information for all kernel developers.
>     fault-injection/index
>     livepatch/index
>     rust/index
> -
> +   test/index

Since you called the directory "tests", this generates a couple of
warnings in the htmldocs build.

(but, again, I think it should be dev-tools/tests, and perhaps
referenced from the selftest docs already there)

jon

Re: [PATCH 0/2] Add support for inlined documentation for kunit and kselftests

[Mauro Carvalho Chehab]

Em Tue, 03 Oct 2023 11:00:20 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab@kernel.org> writes:
> 
> > This is a follow-up of the discussions taken here:
> >
> >    https://lore.kernel.org/linux-doc/20230704132812.02ba97ba@maurocar-mobl2/T/#t
> >
> > I sent a previous version as RFC. This is basically what we had there, with some
> > improvements at test_list.py.
> >
> > It adds a new extension that allows documenting tests using the same tool we're
> > using for DRM unit tests at IGT GPU tools: https://gitlab.freedesktop.org/drm/igt-gpu-tools.
> >
> > While kernel-doc has provided documentation for in-lined functions/struct comments,
> > it was not meant to document tests.
> >
> > Tests need to be grouped by the test functions. It should also be possible to produce
> > other outputs from the documentation, to integrate it with test suites. For instance, 
> > Internally at Intel, we use the comments to generate DOT files hierarchically grouped
> > per feature categories.
> >
> > This is meant to be an initial series to start documenting kunit.  
> 
> I've played with this a bit...a couple of quick impressions:
> 
> - That's quite a chunk of Python code to be adding.  I've not yet had
>   the chance to read it through properly, will hopefully be able to do
>   so soon.  A bit more commenting would not have gone amiss here...

I'll try to add more comments when respin this series.

I guess I should also add a documentation similar to the one I wrote
for IGT [1]:
	https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/blob/master/docs/test_documentation.md?ref_type=heads#documenting-tests-via-testplan

[1] this document is specific for the way IGT uses it; I'll write
    something similar to it considering the names we've agreed for
    KUnit.

> 
> - I kind of think that this should go under dev-tools rather than being
>   a new top-level directory.  Is there a reason not to put it there?

No particular reason. I'll change it to be under dev-tools/tests at
the next submission.

Should I wait for you to take a look at patch 1/2 before sending
a new version?

Regards,
Mauro

Re: [PATCH 0/2] Add support for inlined documentation for kunit and kselftests

[Jonathan Corbet]

Mauro Carvalho Chehab <mchehab@kernel.org> writes:

> Should I wait for you to take a look at patch 1/2 before sending
> a new version?

Go ahead and resend whenever...I'm still digging out from the last few
weeks.

Thanks,

jon