[PATCH v3 1/2] docs: rust: Move testing to a separate page

[PATCH v3 1/2] docs: rust: Move testing to a separate page

[Dirk Behme]

To be able to add more testing documentation move the testing
section to it's own page.

No change on the documentation itself.

Suggested-by: Trevor Gross <tmgross@umich.edu>
Suggested-by: Miguel Ojeda <ojeda@kernel.org>
Reviewed-by: Trevor Gross <tmgross@umich.edu>
Signed-off-by: Dirk Behme <dirk.behme@de.bosch.com>
---
Changes in v3:
 * Add Trevor's Reviewed-by
 * Rebased against v6.8-rc1

 Documentation/rust/general-information.rst | 24 ----------------------
 Documentation/rust/index.rst               |  1 +
 Documentation/rust/testing.rst             | 24 ++++++++++++++++++++++
 3 files changed, 25 insertions(+), 24 deletions(-)
 create mode 100644 Documentation/rust/testing.rst

diff --git a/Documentation/rust/general-information.rst b/Documentation/rust/general-information.rst
index 236c6dd3c647f..081397827a7ea 100644
--- a/Documentation/rust/general-information.rst
+++ b/Documentation/rust/general-information.rst
@@ -77,27 +77,3 @@ configuration:
 	#[cfg(CONFIG_X="y")]   // Enabled as a built-in (`y`)
 	#[cfg(CONFIG_X="m")]   // Enabled as a module   (`m`)
 	#[cfg(not(CONFIG_X))]  // Disabled
-
-
-Testing
--------
-
-There are the tests that come from the examples in the Rust documentation
-and get transformed into KUnit tests. These can be run via KUnit. For example
-via ``kunit_tool`` (``kunit.py``) on the command line::
-
-	./tools/testing/kunit/kunit.py run --make_options LLVM=1 --arch x86_64 --kconfig_add CONFIG_RUST=y
-
-Alternatively, KUnit can run them as kernel built-in at boot. Refer to
-Documentation/dev-tools/kunit/index.rst for the general KUnit documentation
-and Documentation/dev-tools/kunit/architecture.rst for the details of kernel
-built-in vs. command line testing.
-
-Additionally, there are the ``#[test]`` tests. These can be run using
-the ``rusttest`` Make target::
-
-	make LLVM=1 rusttest
-
-This requires the kernel ``.config`` and downloads external repositories.
-It runs the ``#[test]`` tests on the host (currently) and thus is fairly
-limited in what these tests can test.
diff --git a/Documentation/rust/index.rst b/Documentation/rust/index.rst
index 965f2db529e0f..46d35bd395cf5 100644
--- a/Documentation/rust/index.rst
+++ b/Documentation/rust/index.rst
@@ -40,6 +40,7 @@ configurations.
     general-information
     coding-guidelines
     arch-support
+    testing
 
 .. only::  subproject and html
 
diff --git a/Documentation/rust/testing.rst b/Documentation/rust/testing.rst
new file mode 100644
index 0000000000000..ba8a01015abad
--- /dev/null
+++ b/Documentation/rust/testing.rst
@@ -0,0 +1,24 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Testing
+=======
+
+There are the tests that come from the examples in the Rust documentation
+and get transformed into KUnit tests. These can be run via KUnit. For example
+via ``kunit_tool`` (``kunit.py``) on the command line::
+
+	./tools/testing/kunit/kunit.py run --make_options LLVM=1 --arch x86_64 --kconfig_add CONFIG_RUST=y
+
+Alternatively, KUnit can run them as kernel built-in at boot. Refer to
+Documentation/dev-tools/kunit/index.rst for the general KUnit documentation
+and Documentation/dev-tools/kunit/architecture.rst for the details of kernel
+built-in vs. command line testing.
+
+Additionally, there are the ``#[test]`` tests. These can be run using
+the ``rusttest`` Make target::
+
+	make LLVM=1 rusttest
+
+This requires the kernel ``.config`` and downloads external repositories.
+It runs the ``#[test]`` tests on the host (currently) and thus is fairly
+limited in what these tests can test.
-- 
2.28.0

[PATCH v3 2/2] docs: rust: Add description of Rust documentation test as KUnit ones

[Dirk Behme]

Rust documentation tests are automatically converted into KUnit
tests. The commit adding this feature

commit a66d733da801 ("rust: support running Rust documentation tests as KUnit ones")

from Miguel has a very nice commit message with a lot details
for this. To not 'hide' that just in a commit message, pick the main
parts of it and add it to the documentation. And add a short info
how to enable this. While adding this, improve the structure of
the sections.

Co-developed-by: Miguel Ojeda <ojeda@kernel.org>
Signed-off-by: Dirk Behme <dirk.behme@de.bosch.com>
---
Changes in v3:
 * Got the '?' operator link to work :)
 * Rebased against v6.8-rc1

 Documentation/rust/testing.rst | 120 +++++++++++++++++++++++++++++++--
 1 file changed, 116 insertions(+), 4 deletions(-)

diff --git a/Documentation/rust/testing.rst b/Documentation/rust/testing.rst
index ba8a01015abad..4926db9d19592 100644
--- a/Documentation/rust/testing.rst
+++ b/Documentation/rust/testing.rst
@@ -1,11 +1,25 @@
 .. SPDX-License-Identifier: GPL-2.0
 
 Testing
-=======
++++++++
 
-There are the tests that come from the examples in the Rust documentation
-and get transformed into KUnit tests. These can be run via KUnit. For example
-via ``kunit_tool`` (``kunit.py``) on the command line::
+This document contains useful information how to test the Rust code in the kernel.
+
+There are two sorts of tests:
+
+ * The KUnit tests
+ * The ``#[test]`` tests
+
+The KUnit tests
+===============
+
+This are the tests that come from the examples in the Rust documentation
+and get transformed into KUnit tests.
+
+Usage
+~~~~~
+
+These tests can be run via KUnit. For example via ``kunit_tool`` (``kunit.py``) on the command line::
 
 	./tools/testing/kunit/kunit.py run --make_options LLVM=1 --arch x86_64 --kconfig_add CONFIG_RUST=y
 
@@ -14,6 +28,104 @@ Documentation/dev-tools/kunit/index.rst for the general KUnit documentation
 and Documentation/dev-tools/kunit/architecture.rst for the details of kernel
 built-in vs. command line testing.
 
+To use these KUnit doctests, the following must be enabled::
+
+	CONFIG_KUNIT
+	   Kernel hacking -> Kernel Testing and Coverage -> KUnit - Enable support for unit tests
+	CONFIG_RUST_KERNEL_DOCTESTS
+	   Kernel hacking -> Rust hacking -> Doctests for the `kernel` crate
+
+in the kernel config system.
+
+KUnit tests are documentation tests
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+These documentation tests are typically examples of
+usage of any item (e.g. function, struct, module...).
+
+They are very convenient because they are just written
+alongside the documentation. For instance:
+
+.. code-block:: rust
+
+	/// Sums two numbers.
+	///
+	/// ```
+	/// assert_eq!(mymod::f(10, 20), 30);
+	/// ```
+	pub fn f(a: i32, b: i32) -> i32 {
+	    a + b
+	}
+
+In userspace, the tests are collected and run via ``rustdoc``.
+Using the tool as-is would be useful already, since it allows
+verifying that examples compile (thus enforcing they are kept
+in sync with the code they document) and as well as running
+those that do not depend on in-kernel APIs.
+
+For the kernel, however, these tests get transformed into KUnit
+test suites. This means that doctests get compiled as Rust
+kernel objects, allowing them to run against a built kernel.
+
+A benefit of this KUnit integration is that Rust doctests get
+to reuse existing testing facilities. For instance, the kernel
+log would look like::
+
+	KTAP version 1
+	1..1
+	    KTAP version 1
+	    # Subtest: rust_doctests_kernel
+	    1..59
+	    # rust_doctest_kernel_build_assert_rs_0.location: rust/kernel/build_assert.rs:13
+	    ok 1 rust_doctest_kernel_build_assert_rs_0
+	    # rust_doctest_kernel_build_assert_rs_1.location: rust/kernel/build_assert.rs:56
+	    ok 2 rust_doctest_kernel_build_assert_rs_1
+	    # rust_doctest_kernel_init_rs_0.location: rust/kernel/init.rs:122
+	    ok 3 rust_doctest_kernel_init_rs_0
+	    ...
+	    # rust_doctest_kernel_types_rs_2.location: rust/kernel/types.rs:150
+	    ok 59 rust_doctest_kernel_types_rs_2
+	# rust_doctests_kernel: pass:59 fail:0 skip:0 total:59
+	# Totals: pass:59 fail:0 skip:0 total:59
+	ok 1 rust_doctests_kernel
+
+Tests using the `? <https://doc.rust-lang.org/reference/expressions/operator-expr.html#the-question-mark-operator>`_ operator are also supported as usual, e.g.:
+
+.. code-block:: rust
+
+	/// ```
+	/// # use kernel::{spawn_work_item, workqueue};
+	/// spawn_work_item!(workqueue::system(), || pr_info!("x"))?;
+	/// # Ok::<(), Error>(())
+	/// ```
+
+The tests are also compiled with Clippy under ``CLIPPY=1``, just
+like normal code, thus also benefitting from extra linting.
+
+In order for developers to easily see which line of doctest code
+caused a failure, a KTAP diagnostic line is printed to the log.
+This contains the location (file and line) of the original test
+(i.e. instead of the location in the generated Rust file)::
+
+	# rust_doctest_kernel_types_rs_2.location: rust/kernel/types.rs:150
+
+Rust tests appear to assert using the usual ``assert!`` and
+``assert_eq!`` macros from the Rust standard library (``core``).
+We provide a custom version that forwards the call to KUnit instead.
+Importantly, these macros do not require passing context,
+unlike those for KUnit testing (i.e. ``struct kunit *``). This makes
+them easier to use, and readers of the documentation do not need
+to care about which testing framework is used. In addition, it
+may allow us to test third-party code more easily in the future.
+
+A current limitation is that KUnit does not support assertions
+in other tasks. Thus, we presently simply print an error to the
+kernel log if an assertion actually failed. Additionally,
+doctests are not run for nonpublic functions.
+
+The ``#[test]`` tests
+=====================
+
 Additionally, there are the ``#[test]`` tests. These can be run using
 the ``rusttest`` Make target::
 
-- 
2.28.0

Re: [PATCH v3 2/2] docs: rust: Add description of Rust documentation test as KUnit ones

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 6793 bytes --]

On Thu, 25 Jan 2024 at 14:56, Dirk Behme <dirk.behme@de.bosch.com> wrote:
>
> Rust documentation tests are automatically converted into KUnit
> tests. The commit adding this feature
>
> commit a66d733da801 ("rust: support running Rust documentation tests as KUnit ones")
>
> from Miguel has a very nice commit message with a lot details
> for this. To not 'hide' that just in a commit message, pick the main
> parts of it and add it to the documentation. And add a short info
> how to enable this. While adding this, improve the structure of
> the sections.
>
> Co-developed-by: Miguel Ojeda <ojeda@kernel.org>
> Signed-off-by: Dirk Behme <dirk.behme@de.bosch.com>
> ---

Looks good to me. We'll want to update it once we have more general
KUnit bindings (for something beyond just doctests), but this is
excellent for the current state of things.

This is (with Miguel's Signed-off-by, of course),

Reviewed-by: David Gow <davidgow@google.com>

Thanks,
-- David


> Changes in v3:
>  * Got the '?' operator link to work :)
>  * Rebased against v6.8-rc1
>
>  Documentation/rust/testing.rst | 120 +++++++++++++++++++++++++++++++--
>  1 file changed, 116 insertions(+), 4 deletions(-)
>
> diff --git a/Documentation/rust/testing.rst b/Documentation/rust/testing.rst
> index ba8a01015abad..4926db9d19592 100644
> --- a/Documentation/rust/testing.rst
> +++ b/Documentation/rust/testing.rst
> @@ -1,11 +1,25 @@
>  .. SPDX-License-Identifier: GPL-2.0
>
>  Testing
> -=======
> ++++++++
>
> -There are the tests that come from the examples in the Rust documentation
> -and get transformed into KUnit tests. These can be run via KUnit. For example
> -via ``kunit_tool`` (``kunit.py``) on the command line::
> +This document contains useful information how to test the Rust code in the kernel.
> +
> +There are two sorts of tests:
> +
> + * The KUnit tests
> + * The ``#[test]`` tests
> +
> +The KUnit tests
> +===============
> +
> +This are the tests that come from the examples in the Rust documentation
> +and get transformed into KUnit tests.
> +
> +Usage
> +~~~~~
> +
> +These tests can be run via KUnit. For example via ``kunit_tool`` (``kunit.py``) on the command line::
>
>         ./tools/testing/kunit/kunit.py run --make_options LLVM=1 --arch x86_64 --kconfig_add CONFIG_RUST=y
>
> @@ -14,6 +28,104 @@ Documentation/dev-tools/kunit/index.rst for the general KUnit documentation
>  and Documentation/dev-tools/kunit/architecture.rst for the details of kernel
>  built-in vs. command line testing.
>
> +To use these KUnit doctests, the following must be enabled::
> +
> +       CONFIG_KUNIT
> +          Kernel hacking -> Kernel Testing and Coverage -> KUnit - Enable support for unit tests
> +       CONFIG_RUST_KERNEL_DOCTESTS
> +          Kernel hacking -> Rust hacking -> Doctests for the `kernel` crate
> +
> +in the kernel config system.
> +
> +KUnit tests are documentation tests
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +These documentation tests are typically examples of
> +usage of any item (e.g. function, struct, module...).
> +
> +They are very convenient because they are just written
> +alongside the documentation. For instance:
> +
> +.. code-block:: rust
> +
> +       /// Sums two numbers.
> +       ///
> +       /// ```
> +       /// assert_eq!(mymod::f(10, 20), 30);
> +       /// ```
> +       pub fn f(a: i32, b: i32) -> i32 {
> +           a + b
> +       }
> +
> +In userspace, the tests are collected and run via ``rustdoc``.
> +Using the tool as-is would be useful already, since it allows
> +verifying that examples compile (thus enforcing they are kept
> +in sync with the code they document) and as well as running
> +those that do not depend on in-kernel APIs.
> +
> +For the kernel, however, these tests get transformed into KUnit
> +test suites. This means that doctests get compiled as Rust
> +kernel objects, allowing them to run against a built kernel.
> +
> +A benefit of this KUnit integration is that Rust doctests get
> +to reuse existing testing facilities. For instance, the kernel
> +log would look like::
> +
> +       KTAP version 1
> +       1..1
> +           KTAP version 1
> +           # Subtest: rust_doctests_kernel
> +           1..59
> +           # rust_doctest_kernel_build_assert_rs_0.location: rust/kernel/build_assert.rs:13
> +           ok 1 rust_doctest_kernel_build_assert_rs_0
> +           # rust_doctest_kernel_build_assert_rs_1.location: rust/kernel/build_assert.rs:56
> +           ok 2 rust_doctest_kernel_build_assert_rs_1
> +           # rust_doctest_kernel_init_rs_0.location: rust/kernel/init.rs:122
> +           ok 3 rust_doctest_kernel_init_rs_0
> +           ...
> +           # rust_doctest_kernel_types_rs_2.location: rust/kernel/types.rs:150
> +           ok 59 rust_doctest_kernel_types_rs_2
> +       # rust_doctests_kernel: pass:59 fail:0 skip:0 total:59
> +       # Totals: pass:59 fail:0 skip:0 total:59
> +       ok 1 rust_doctests_kernel
> +
> +Tests using the `? <https://doc.rust-lang.org/reference/expressions/operator-expr.html#the-question-mark-operator>`_ operator are also supported as usual, e.g.:
> +
> +.. code-block:: rust
> +
> +       /// ```
> +       /// # use kernel::{spawn_work_item, workqueue};
> +       /// spawn_work_item!(workqueue::system(), || pr_info!("x"))?;
> +       /// # Ok::<(), Error>(())
> +       /// ```
> +
> +The tests are also compiled with Clippy under ``CLIPPY=1``, just
> +like normal code, thus also benefitting from extra linting.
> +
> +In order for developers to easily see which line of doctest code
> +caused a failure, a KTAP diagnostic line is printed to the log.
> +This contains the location (file and line) of the original test
> +(i.e. instead of the location in the generated Rust file)::
> +
> +       # rust_doctest_kernel_types_rs_2.location: rust/kernel/types.rs:150
> +
> +Rust tests appear to assert using the usual ``assert!`` and
> +``assert_eq!`` macros from the Rust standard library (``core``).
> +We provide a custom version that forwards the call to KUnit instead.
> +Importantly, these macros do not require passing context,
> +unlike those for KUnit testing (i.e. ``struct kunit *``). This makes
> +them easier to use, and readers of the documentation do not need
> +to care about which testing framework is used. In addition, it
> +may allow us to test third-party code more easily in the future.
> +
> +A current limitation is that KUnit does not support assertions
> +in other tasks. Thus, we presently simply print an error to the
> +kernel log if an assertion actually failed. Additionally,
> +doctests are not run for nonpublic functions.
> +
> +The ``#[test]`` tests
> +=====================
> +
>  Additionally, there are the ``#[test]`` tests. These can be run using
>  the ``rusttest`` Make target::
>
> --
> 2.28.0
>
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4014 bytes --]

Re: [PATCH v3 2/2] docs: rust: Add description of Rust documentation test as KUnit ones

[Alice Ryhl]

> Rust documentation tests are automatically converted into KUnit
> tests. The commit adding this feature
> 
> commit a66d733da801 ("rust: support running Rust documentation tests as KUnit ones")
> 
> from Miguel has a very nice commit message with a lot details
> for this. To not 'hide' that just in a commit message, pick the main
> parts of it and add it to the documentation. And add a short info
> how to enable this. While adding this, improve the structure of
> the sections.
> 
> Co-developed-by: Miguel Ojeda <ojeda@kernel.org>
> Signed-off-by: Dirk Behme <dirk.behme@de.bosch.com>

Reviewed-by: Alice Ryhl <aliceryhl@google.com>

Re: [PATCH v3 2/2] docs: rust: Add description of Rust documentation test as KUnit ones

[Miguel Ojeda]

On Thu, Jan 25, 2024 at 7:56 AM Dirk Behme <dirk.behme@de.bosch.com> wrote:
>
> Changes in v3:
>  * Got the '?' operator link to work :)
>  * Rebased against v6.8-rc1

Great, thanks for both :)

> +This are the tests that come from the examples in the Rust documentation
> +and get transformed into KUnit tests.

Typo: "This".

I would also perhaps say "...documentation. They get transformed..."
instead so that it is clear there is not another set of doctests that
are not transformed.

Cheers,
Miguel

Re: [PATCH v3 1/2] docs: rust: Move testing to a separate page

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 3998 bytes --]

On Thu, 25 Jan 2024 at 14:56, Dirk Behme <dirk.behme@de.bosch.com> wrote:
>
> To be able to add more testing documentation move the testing
> section to it's own page.
>
> No change on the documentation itself.
>
> Suggested-by: Trevor Gross <tmgross@umich.edu>
> Suggested-by: Miguel Ojeda <ojeda@kernel.org>
> Reviewed-by: Trevor Gross <tmgross@umich.edu>
> Signed-off-by: Dirk Behme <dirk.behme@de.bosch.com>
> ---
> Changes in v3:
>  * Add Trevor's Reviewed-by
>  * Rebased against v6.8-rc1
>

Makes sense to me. Thanks for rebasing it, too: made reviewing it much easier.

Reviewed-by: David Gow <davidgow@google.com>

Cheers,
-- David



>  Documentation/rust/general-information.rst | 24 ----------------------
>  Documentation/rust/index.rst               |  1 +
>  Documentation/rust/testing.rst             | 24 ++++++++++++++++++++++
>  3 files changed, 25 insertions(+), 24 deletions(-)
>  create mode 100644 Documentation/rust/testing.rst
>
> diff --git a/Documentation/rust/general-information.rst b/Documentation/rust/general-information.rst
> index 236c6dd3c647f..081397827a7ea 100644
> --- a/Documentation/rust/general-information.rst
> +++ b/Documentation/rust/general-information.rst
> @@ -77,27 +77,3 @@ configuration:
>         #[cfg(CONFIG_X="y")]   // Enabled as a built-in (`y`)
>         #[cfg(CONFIG_X="m")]   // Enabled as a module   (`m`)
>         #[cfg(not(CONFIG_X))]  // Disabled
> -
> -
> -Testing
> --------
> -
> -There are the tests that come from the examples in the Rust documentation
> -and get transformed into KUnit tests. These can be run via KUnit. For example
> -via ``kunit_tool`` (``kunit.py``) on the command line::
> -
> -       ./tools/testing/kunit/kunit.py run --make_options LLVM=1 --arch x86_64 --kconfig_add CONFIG_RUST=y
> -
> -Alternatively, KUnit can run them as kernel built-in at boot. Refer to
> -Documentation/dev-tools/kunit/index.rst for the general KUnit documentation
> -and Documentation/dev-tools/kunit/architecture.rst for the details of kernel
> -built-in vs. command line testing.
> -
> -Additionally, there are the ``#[test]`` tests. These can be run using
> -the ``rusttest`` Make target::
> -
> -       make LLVM=1 rusttest
> -
> -This requires the kernel ``.config`` and downloads external repositories.
> -It runs the ``#[test]`` tests on the host (currently) and thus is fairly
> -limited in what these tests can test.
> diff --git a/Documentation/rust/index.rst b/Documentation/rust/index.rst
> index 965f2db529e0f..46d35bd395cf5 100644
> --- a/Documentation/rust/index.rst
> +++ b/Documentation/rust/index.rst
> @@ -40,6 +40,7 @@ configurations.
>      general-information
>      coding-guidelines
>      arch-support
> +    testing
>
>  .. only::  subproject and html
>
> diff --git a/Documentation/rust/testing.rst b/Documentation/rust/testing.rst
> new file mode 100644
> index 0000000000000..ba8a01015abad
> --- /dev/null
> +++ b/Documentation/rust/testing.rst
> @@ -0,0 +1,24 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +Testing
> +=======
> +
> +There are the tests that come from the examples in the Rust documentation
> +and get transformed into KUnit tests. These can be run via KUnit. For example
> +via ``kunit_tool`` (``kunit.py``) on the command line::
> +
> +       ./tools/testing/kunit/kunit.py run --make_options LLVM=1 --arch x86_64 --kconfig_add CONFIG_RUST=y
> +
> +Alternatively, KUnit can run them as kernel built-in at boot. Refer to
> +Documentation/dev-tools/kunit/index.rst for the general KUnit documentation
> +and Documentation/dev-tools/kunit/architecture.rst for the details of kernel
> +built-in vs. command line testing.
> +
> +Additionally, there are the ``#[test]`` tests. These can be run using
> +the ``rusttest`` Make target::
> +
> +       make LLVM=1 rusttest
> +
> +This requires the kernel ``.config`` and downloads external repositories.
> +It runs the ``#[test]`` tests on the host (currently) and thus is fairly
> +limited in what these tests can test.
> --
> 2.28.0
>
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4014 bytes --]

Re: [PATCH v3 1/2] docs: rust: Move testing to a separate page

[Alice Ryhl]

> To be able to add more testing documentation move the testing
> section to it's own page.
> 
> No change on the documentation itself.
> 
> Suggested-by: Trevor Gross <tmgross@umich.edu>
> Suggested-by: Miguel Ojeda <ojeda@kernel.org>
> Reviewed-by: Trevor Gross <tmgross@umich.edu>
> Signed-off-by: Dirk Behme <dirk.behme@de.bosch.com>

Reviewed-by: Alice Ryhl <aliceryhl@google.com>