[igt-dev] [RFC] IGT Subtest Documentation

[igt-dev] [RFC] IGT Subtest Documentation

[Arkadiusz Hiler]

## The Idea

Most of the tests are quite easy to follow but yet it's hard to tell why they
are doing what they are doing if you are not the author. Their names are just
not enough.

That results in archaeological diggings using `git blame` and mailing list
archives just to grasp the purpose. Commit history gets diluted over time as
improvements and refactors amass, making it progressively harder to find the
original intention.

**The idea** is to provide concise commit-message-quality explanation along
the code, and make the code more readable and available as a part of
documentation.

"Describe the spirit of the test, not the letter."


### Why not more detailed descriptions?

Translating C to English is a bad idea, as it effectively duplicates the
code. Both *implementations* will diverge and increase the maintenance
burden.

If parts of code are hard to follow or do not ready easy enough they should
be refactored and/or annotated with normal C comments.

Any extra clarifications, diagrams, etc. should be added as free-form
comments to the code, which will be accessible through documentation.

If someone is unable to read *annotated* IGT source code, English transpill
will not help.


### What are the benefits of this approach?

1. The descriptions available on command line via
   `--describe [PATTERN]`

2. Generated docs will contain the test name with description for an easy
   online lookup. The doc will also link to source file on the exact line of
   the subtest for quick reference.


## The C API

```c
igt_describe_test(char *dsc);

igt_describe(char *dsc);
igt_describe_f(char *fmt, ...);
```


### Example Code

```c
/* tests/frob_knob_basic.c */
#include "igt.h"
#include "frob.h"

igt_describe_test("Check basic functionality of the frob IOCTL on a knob");
igt_main
{
	igt_describe("Basic test making sure that the frobbing IOCTL "
		     "succeeds on the given knob with no arguments.\n"):
	igt_subtest_group {
		igt_subtest("frob-the-knob-a")
			test_frob(KNOB_A);

		igt_subtest("frob-the-knob-b")
			test_frob(KNOB_B);
	}

	igt_describe("Frobbing and Skewing are used by asynchronous outer "
		     "space, so one should never stall on the other."):
	igt_subtest_group {
		igt_describe("Make sure that frob does not stall for a skew by "
			     "squeezing many frobs in-between two skews.\n"):
		igt_subtest("frob-vs-skew")
			test_frob_vs_skew();

		igt_describe("Make sure that skew does not stall for a frob "
			     "by doing many skews and frobs simultaneously "
			     "and making sure that we maintain sensible "
			     "throughput of each.\n"):
		igt_subtest("skew-vs-frob");
			test_skew_vs_frob();
	}
}
```


### Example Output

```
$ ./frob_knob_basic --describe

Check basic functionality of the frob IOCT on a knob.

SUB frob-the-knob-a:
  Basic test making sure that the frobbing IOCTL succeed on the given knob
  with no arguments.

SUB frob-the-knob-b:
  Basic test making sure that the frobbing IOCTL succeed on the given knob
  with no arguments.

SUB frob-vs-skew:
  Frobbing and Skewing are used by asynchronous outer space, so one should
  never stall on the other.

  Make sure that frob does not stall for a skew by squeezing many frobs
  in-between two skews.

SUB skew-vs-frob:
  Frobbing and Skewing are used by asynchronous outer space, so one should
  never stall on the other.

  Make sure that skew does not stall for a frob by doing many skews and frobs
  simultaneously and making sure that we maintain sensible throughput of
  each.
```


### Generated Documentation

We are already generating dcoumnetation by invoking `--list-subtests` and
`--help-description`. We can easily extend that to use the new `--describe`.

With a bit of gymnastics we can pull the whole `.c` file into the
documentation and link subtests to the line they are introduced on (either in
the local copy of the source or in the gitlab).

This needs small extensions to the --describe format as follows:

```
SUB frob-the-knob-a test/frob-knob-basic.c:9:
  ...
```


## FAQ

**Q**: `igt_describe_test()` before main? How?
**A**: it's a rename of `IGT_TEST_DESCRIPTION()` macro we already use.

**Q**: I've seen that somewhere before...
**A**: https://patchwork.freedesktop.org/patch/171220/

**Q**: What has happened with the other implementation mentioned by Daniel?
**A**: I was unable to find more details on it. Seems like it has not emerged.
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [RFC] IGT Subtest Documentation

[Daniel Vetter]

On Tue, May 28, 2019 at 03:29:34PM +0300, Arkadiusz Hiler wrote:
> ## The Idea
> 
> Most of the tests are quite easy to follow but yet it's hard to tell why they
> are doing what they are doing if you are not the author. Their names are just
> not enough.
> 
> That results in archaeological diggings using `git blame` and mailing list
> archives just to grasp the purpose. Commit history gets diluted over time as
> improvements and refactors amass, making it progressively harder to find the
> original intention.
> 
> **The idea** is to provide concise commit-message-quality explanation along
> the code, and make the code more readable and available as a part of
> documentation.
> 
> "Describe the spirit of the test, not the letter."
> 
> 
> ### Why not more detailed descriptions?
> 
> Translating C to English is a bad idea, as it effectively duplicates the
> code. Both *implementations* will diverge and increase the maintenance
> burden.
> 
> If parts of code are hard to follow or do not ready easy enough they should
> be refactored and/or annotated with normal C comments.
> 
> Any extra clarifications, diagrams, etc. should be added as free-form
> comments to the code, which will be accessible through documentation.
> 
> If someone is unable to read *annotated* IGT source code, English transpill
> will not help.
> 
> 
> ### What are the benefits of this approach?
> 
> 1. The descriptions available on command line via
>    `--describe [PATTERN]`
> 
> 2. Generated docs will contain the test name with description for an easy
>    online lookup. The doc will also link to source file on the exact line of
>    the subtest for quick reference.
> 
> 
> ## The C API
> 
> ```c
> igt_describe_test(char *dsc);
> 
> igt_describe(char *dsc);
> igt_describe_f(char *fmt, ...);
> ```
> 
> 
> ### Example Code
> 
> ```c
> /* tests/frob_knob_basic.c */
> #include "igt.h"
> #include "frob.h"
> 
> igt_describe_test("Check basic functionality of the frob IOCTL on a knob");
> igt_main
> {
> 	igt_describe("Basic test making sure that the frobbing IOCTL "
> 		     "succeeds on the given knob with no arguments.\n"):
> 	igt_subtest_group {
> 		igt_subtest("frob-the-knob-a")
> 			test_frob(KNOB_A);
> 
> 		igt_subtest("frob-the-knob-b")
> 			test_frob(KNOB_B);
> 	}
> 
> 	igt_describe("Frobbing and Skewing are used by asynchronous outer "
> 		     "space, so one should never stall on the other."):
> 	igt_subtest_group {
> 		igt_describe("Make sure that frob does not stall for a skew by "
> 			     "squeezing many frobs in-between two skews.\n"):
> 		igt_subtest("frob-vs-skew")
> 			test_frob_vs_skew();
> 
> 		igt_describe("Make sure that skew does not stall for a frob "
> 			     "by doing many skews and frobs simultaneously "
> 			     "and making sure that we maintain sensible "
> 			     "throughput of each.\n"):
> 		igt_subtest("skew-vs-frob");
> 			test_skew_vs_frob();
> 	}
> }
> ```
> 
> 
> ### Example Output
> 
> ```
> $ ./frob_knob_basic --describe
> 
> Check basic functionality of the frob IOCT on a knob.
> 
> SUB frob-the-knob-a:
>   Basic test making sure that the frobbing IOCTL succeed on the given knob
>   with no arguments.
> 
> SUB frob-the-knob-b:
>   Basic test making sure that the frobbing IOCTL succeed on the given knob
>   with no arguments.
> 
> SUB frob-vs-skew:
>   Frobbing and Skewing are used by asynchronous outer space, so one should
>   never stall on the other.
> 
>   Make sure that frob does not stall for a skew by squeezing many frobs
>   in-between two skews.
> 
> SUB skew-vs-frob:
>   Frobbing and Skewing are used by asynchronous outer space, so one should
>   never stall on the other.
> 
>   Make sure that skew does not stall for a frob by doing many skews and frobs
>   simultaneously and making sure that we maintain sensible throughput of
>   each.
> ```
> 
> 
> ### Generated Documentation
> 
> We are already generating dcoumnetation by invoking `--list-subtests` and
> `--help-description`. We can easily extend that to use the new `--describe`.
> 
> With a bit of gymnastics we can pull the whole `.c` file into the
> documentation and link subtests to the line they are introduced on (either in
> the local copy of the source or in the gitlab).
> 
> This needs small extensions to the --describe format as follows:
> 
> ```
> SUB frob-the-knob-a test/frob-knob-basic.c:9:
>   ...
> ```
> 
> 
> ## FAQ
> 
> **Q**: `igt_describe_test()` before main? How?
> **A**: it's a rename of `IGT_TEST_DESCRIPTION()` macro we already use.
> 
> **Q**: I've seen that somewhere before...
> **A**: https://patchwork.freedesktop.org/patch/171220/
> 
> **Q**: What has happened with the other implementation mentioned by Daniel?
> **A**: I was unable to find more details on it. Seems like it has not emerged.

Feels like a natural extension of IGT_TEST_DESCRIPTION, and given how
little uptake that had I don't think we should aim for more. Also, better
chance that people won't just reword what the C code does in bad English,
but actually try to explain the aim of a testcase.

+1 me

Yes that's a complete flip from me, but the gtkdoc thing died and no one
cares maybe this gets things started. I still think the
IGT_TEST_DESCRIPTION stuff is rather hard on the eyes :-)

Cheers, Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [RFC] IGT Subtest Documentation

[Ser, Simon]

On Tue, 2019-05-28 at 15:29 +0300, Arkadiusz Hiler wrote:
> ## The Idea
> 
> Most of the tests are quite easy to follow but yet it's hard to tell why they
> are doing what they are doing if you are not the author. Their names are just
> not enough.
> 
> That results in archaeological diggings using `git blame` and mailing list
> archives just to grasp the purpose. Commit history gets diluted over time as
> improvements and refactors amass, making it progressively harder to find the
> original intention.
> 
> **The idea** is to provide concise commit-message-quality explanation along
> the code, and make the code more readable and available as a part of
> documentation.
> 
> "Describe the spirit of the test, not the letter."
> 
> 
> ### Why not more detailed descriptions?
> 
> Translating C to English is a bad idea, as it effectively duplicates the
> code. Both *implementations* will diverge and increase the maintenance
> burden.
> 
> If parts of code are hard to follow or do not ready easy enough they should
> be refactored and/or annotated with normal C comments.
> 
> Any extra clarifications, diagrams, etc. should be added as free-form
> comments to the code, which will be accessible through documentation.
> 
> If someone is unable to read *annotated* IGT source code, English transpill
> will not help.
> 
> 
> ### What are the benefits of this approach?
> 
> 1. The descriptions available on command line via
>    `--describe [PATTERN]`
> 
> 2. Generated docs will contain the test name with description for an easy
>    online lookup. The doc will also link to source file on the exact line of
>    the subtest for quick reference.
> 
> 
> ## The C API
> 
> ```c
> igt_describe_test(char *dsc);
> 
> igt_describe(char *dsc);
> igt_describe_f(char *fmt, ...);
> ```
> 
> 
> ### Example Code
> 
> ```c
> /* tests/frob_knob_basic.c */
> #include "igt.h"
> #include "frob.h"
> 
> igt_describe_test("Check basic functionality of the frob IOCTL on a knob");
> igt_main
> {
> 	igt_describe("Basic test making sure that the frobbing IOCTL "
> 		     "succeeds on the given knob with no arguments.\n"):
> 	igt_subtest_group {
> 		igt_subtest("frob-the-knob-a")
> 			test_frob(KNOB_A);
> 
> 		igt_subtest("frob-the-knob-b")
> 			test_frob(KNOB_B);
> 	}
> 
> 	igt_describe("Frobbing and Skewing are used by asynchronous outer "
> 		     "space, so one should never stall on the other."):
> 	igt_subtest_group {
> 		igt_describe("Make sure that frob does not stall for a skew by "
> 			     "squeezing many frobs in-between two skews.\n"):
> 		igt_subtest("frob-vs-skew")
> 			test_frob_vs_skew();
> 
> 		igt_describe("Make sure that skew does not stall for a frob "
> 			     "by doing many skews and frobs simultaneously "
> 			     "and making sure that we maintain sensible "
> 			     "throughput of each.\n"):
> 		igt_subtest("skew-vs-frob");
> 			test_skew_vs_frob();
> 	}
> }
> ```
> 
> 
> ### Example Output
> 
> ```
> $ ./frob_knob_basic --describe
> 
> Check basic functionality of the frob IOCT on a knob.
> 
> SUB frob-the-knob-a:
>   Basic test making sure that the frobbing IOCTL succeed on the given knob
>   with no arguments.
> 
> SUB frob-the-knob-b:
>   Basic test making sure that the frobbing IOCTL succeed on the given knob
>   with no arguments.
> 
> SUB frob-vs-skew:
>   Frobbing and Skewing are used by asynchronous outer space, so one should
>   never stall on the other.
> 
>   Make sure that frob does not stall for a skew by squeezing many frobs
>   in-between two skews.
> 
> SUB skew-vs-frob:
>   Frobbing and Skewing are used by asynchronous outer space, so one should
>   never stall on the other.
> 
>   Make sure that skew does not stall for a frob by doing many skews and frobs
>   simultaneously and making sure that we maintain sensible throughput of
>   each.
> ```
> 
> 
> ### Generated Documentation
> 
> We are already generating dcoumnetation by invoking `--list-subtests` and
> `--help-description`. We can easily extend that to use the new `--describe`.
> 
> With a bit of gymnastics we can pull the whole `.c` file into the
> documentation and link subtests to the line they are introduced on (either in
> the local copy of the source or in the gitlab).
> 
> This needs small extensions to the --describe format as follows:
> 
> ```
> SUB frob-the-knob-a test/frob-knob-basic.c:9:
>   ...
> ```
> 
> 
> ## FAQ
> 
> **Q**: `igt_describe_test()` before main? How?
> **A**: it's a rename of `IGT_TEST_DESCRIPTION()` macro we already use.
> 
> **Q**: I've seen that somewhere before...
> **A**: https://patchwork.freedesktop.org/patch/171220/
> 
> **Q**: What has happened with the other implementation mentioned by Daniel?
> **A**: I was unable to find more details on it. Seems like it has not emerged.

LGTM. It's important that these descriptions stay small.

> _______________________________________________
> igt-dev mailing list
> igt-dev@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/igt-dev
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [RFC] IGT Subtest Documentation

[Katarzyna Dec]

On Tue, May 28, 2019 at 03:29:34PM +0300, Arkadiusz Hiler wrote:
> ## The Idea
> 
> Most of the tests are quite easy to follow but yet it's hard to tell why they
> are doing what they are doing if you are not the author. Their names are just
> not enough.
> 
> That results in archaeological diggings using `git blame` and mailing list
> archives just to grasp the purpose. Commit history gets diluted over time as
> improvements and refactors amass, making it progressively harder to find the
> original intention.
> 
> **The idea** is to provide concise commit-message-quality explanation along
> the code, and make the code more readable and available as a part of
> documentation.
> 
> "Describe the spirit of the test, not the letter."
> 
> 
> ### Why not more detailed descriptions?
> 
> Translating C to English is a bad idea, as it effectively duplicates the
> code. Both *implementations* will diverge and increase the maintenance
> burden.
> 
> If parts of code are hard to follow or do not ready easy enough they should
> be refactored and/or annotated with normal C comments.
> 
> Any extra clarifications, diagrams, etc. should be added as free-form
> comments to the code, which will be accessible through documentation.
> 
> If someone is unable to read *annotated* IGT source code, English transpill
> will not help.
> 
> 
> ### What are the benefits of this approach?
> 
> 1. The descriptions available on command line via
>    `--describe [PATTERN]`
> 
> 2. Generated docs will contain the test name with description for an easy
>    online lookup. The doc will also link to source file on the exact line of
>    the subtest for quick reference.
> 
> 
> ## The C API
> 
> ```c
> igt_describe_test(char *dsc);
> 
> igt_describe(char *dsc);
> igt_describe_f(char *fmt, ...);
> ```
> 
> 
> ### Example Code
> 
> ```c
> /* tests/frob_knob_basic.c */
> #include "igt.h"
> #include "frob.h"
> 
> igt_describe_test("Check basic functionality of the frob IOCTL on a knob");
> igt_main
> {
> 	igt_describe("Basic test making sure that the frobbing IOCTL "
> 		     "succeeds on the given knob with no arguments.\n"):
> 	igt_subtest_group {
> 		igt_subtest("frob-the-knob-a")
> 			test_frob(KNOB_A);
> 
> 		igt_subtest("frob-the-knob-b")
> 			test_frob(KNOB_B);
> 	}
> 
> 	igt_describe("Frobbing and Skewing are used by asynchronous outer "
> 		     "space, so one should never stall on the other."):
> 	igt_subtest_group {
> 		igt_describe("Make sure that frob does not stall for a skew by "
> 			     "squeezing many frobs in-between two skews.\n"):
> 		igt_subtest("frob-vs-skew")
> 			test_frob_vs_skew();
> 
> 		igt_describe("Make sure that skew does not stall for a frob "
> 			     "by doing many skews and frobs simultaneously "
> 			     "and making sure that we maintain sensible "
> 			     "throughput of each.\n"):
> 		igt_subtest("skew-vs-frob");
> 			test_skew_vs_frob();
> 	}
> }
> ```
> 
> 
> ### Example Output
> 
> ```
> $ ./frob_knob_basic --describe
> 
> Check basic functionality of the frob IOCT on a knob.
> 
> SUB frob-the-knob-a:
>   Basic test making sure that the frobbing IOCTL succeed on the given knob
>   with no arguments.
> 
> SUB frob-the-knob-b:
>   Basic test making sure that the frobbing IOCTL succeed on the given knob
>   with no arguments.
> 
> SUB frob-vs-skew:
>   Frobbing and Skewing are used by asynchronous outer space, so one should
>   never stall on the other.
> 
>   Make sure that frob does not stall for a skew by squeezing many frobs
>   in-between two skews.
> 
> SUB skew-vs-frob:
>   Frobbing and Skewing are used by asynchronous outer space, so one should
>   never stall on the other.
> 
>   Make sure that skew does not stall for a frob by doing many skews and frobs
>   simultaneously and making sure that we maintain sensible throughput of
>   each.
> ```
> 
> 
> ### Generated Documentation
> 
> We are already generating dcoumnetation by invoking `--list-subtests` and
> `--help-description`. We can easily extend that to use the new `--describe`.
> 
> With a bit of gymnastics we can pull the whole `.c` file into the
> documentation and link subtests to the line they are introduced on (either in
> the local copy of the source or in the gitlab).
> 
> This needs small extensions to the --describe format as follows:
> 
> ```
> SUB frob-the-knob-a test/frob-knob-basic.c:9:
>   ...
> ```
> 
> 
> ## FAQ
> 
> **Q**: `igt_describe_test()` before main? How?
> **A**: it's a rename of `IGT_TEST_DESCRIPTION()` macro we already use.
> 
> **Q**: I've seen that somewhere before...
> **A**: https://patchwork.freedesktop.org/patch/171220/
> 
> **Q**: What has happened with the other implementation mentioned by Daniel?
> **A**: I was unable to find more details on it. Seems like it has not emerged.

Wow Arek! That sound awsome. I hope that something in that shape will be
introduced to igt and respected.
Do we want somehow enforce on adding docs? How deep in description we should be?
Like - this test is 'doing reset' or is checking HW/SW feature?

Kasia

> _______________________________________________
> igt-dev mailing list
> igt-dev@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/igt-dev
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [RFC] IGT Subtest Documentation

[Kalamarz, Lukasz]

On Tue, 2019-05-28 at 15:29 +0300, Arkadiusz Hiler wrote:
> ## The Idea
> 
> Most of the tests are quite easy to follow but yet it's hard to tell
> why they
> are doing what they are doing if you are not the author. Their names
> are just
> not enough.
> 
> That results in archaeological diggings using `git blame` and mailing
> list
> archives just to grasp the purpose. Commit history gets diluted over
> time as
> improvements and refactors amass, making it progressively harder to
> find the
> original intention.
> 
> **The idea** is to provide concise commit-message-quality explanation
> along
> the code, and make the code more readable and available as a part of
> documentation.
> 
> "Describe the spirit of the test, not the letter."
> 
> 
> ### Why not more detailed descriptions?
> 
> Translating C to English is a bad idea, as it effectively duplicates
> the
> code. Both *implementations* will diverge and increase the
> maintenance
> burden.
> 
> If parts of code are hard to follow or do not ready easy enough they
> should
> be refactored and/or annotated with normal C comments.
> 
> Any extra clarifications, diagrams, etc. should be added as free-form
> comments to the code, which will be accessible through documentation.
> 
> If someone is unable to read *annotated* IGT source code, English
> transpill
> will not help.
> 
> 
> ### What are the benefits of this approach?
> 
> 1. The descriptions available on command line via
>    `--describe [PATTERN]`
> 
> 2. Generated docs will contain the test name with description for an
> easy
>    online lookup. The doc will also link to source file on the exact
> line of
>    the subtest for quick reference.
> 
> 
> ## The C API
> 
> ```c
> igt_describe_test(char *dsc);
> 
> igt_describe(char *dsc);
> igt_describe_f(char *fmt, ...);
> ```
> 
> 
> ### Example Code
> 
> ```c
> /* tests/frob_knob_basic.c */
> #include "igt.h"
> #include "frob.h"
> 
> igt_describe_test("Check basic functionality of the frob IOCTL on a
> knob");
> igt_main
> {
> 	igt_describe("Basic test making sure that the frobbing IOCTL "
> 		     "succeeds on the given knob with no
> arguments.\n"):
> 	igt_subtest_group {
> 		igt_subtest("frob-the-knob-a")
> 			test_frob(KNOB_A);
> 
> 		igt_subtest("frob-the-knob-b")
> 			test_frob(KNOB_B);
> 	}
> 
> 	igt_describe("Frobbing and Skewing are used by asynchronous
> outer "
> 		     "space, so one should never stall on the other."):
> 	igt_subtest_group {
> 		igt_describe("Make sure that frob does not stall for a
> skew by "
> 			     "squeezing many frobs in-between two
> skews.\n"):
> 		igt_subtest("frob-vs-skew")
> 			test_frob_vs_skew();
> 
> 		igt_describe("Make sure that skew does not stall for a
> frob "
> 			     "by doing many skews and frobs
> simultaneously "
> 			     "and making sure that we maintain sensible
> "
> 			     "throughput of each.\n"):
> 		igt_subtest("skew-vs-frob");
> 			test_skew_vs_frob();
> 	}
> }
> ```
> 
> 
> ### Example Output
> 
> ```
> $ ./frob_knob_basic --describe
> 
> Check basic functionality of the frob IOCT on a knob.
> 
> SUB frob-the-knob-a:
>   Basic test making sure that the frobbing IOCTL succeed on the given
> knob
>   with no arguments.
> 
> SUB frob-the-knob-b:
>   Basic test making sure that the frobbing IOCTL succeed on the given
> knob
>   with no arguments.
> 
> SUB frob-vs-skew:
>   Frobbing and Skewing are used by asynchronous outer space, so one
> should
>   never stall on the other.
> 
>   Make sure that frob does not stall for a skew by squeezing many
> frobs
>   in-between two skews.
> 
> SUB skew-vs-frob:
>   Frobbing and Skewing are used by asynchronous outer space, so one
> should
>   never stall on the other.
> 
>   Make sure that skew does not stall for a frob by doing many skews
> and frobs
>   simultaneously and making sure that we maintain sensible throughput
> of
>   each.
> ```
> 
> 
> ### Generated Documentation
> 
> We are already generating dcoumnetation by invoking `--list-subtests` 
> and
> `--help-description`. We can easily extend that to use the new `
> --describe`.
> 
> With a bit of gymnastics we can pull the whole `.c` file into the
> documentation and link subtests to the line they are introduced on
> (either in
> the local copy of the source or in the gitlab).
> 
> This needs small extensions to the --describe format as follows:
> 
> ```
> SUB frob-the-knob-a test/frob-knob-basic.c:9:
>   ...
> ```

LGTM:) Had one question. With test description that will include
reasoning for a given testcase/binary, should it be duplicated in
commit message?
----
Lukasz

> 
> 
> ## FAQ
> 
> **Q**: `igt_describe_test()` before main? How?
> **A**: it's a rename of `IGT_TEST_DESCRIPTION()` macro we already
> use.
> 
> **Q**: I've seen that somewhere before...
> **A**: https://patchwork.freedesktop.org/patch/171220/
> 
> **Q**: What has happened with the other implementation mentioned by
> Daniel?
> **A**: I was unable to find more details on it. Seems like it has not
> emerged.
> _______________________________________________
> igt-dev mailing list
> igt-dev@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/igt-dev
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [RFC] IGT Subtest Documentation

[Arkadiusz Hiler]

On Mon, Jun 03, 2019 at 04:51:46PM +0200, Katarzyna Dec wrote:
> Wow Arek! That sound awsome. I hope that something in that shape will be
> introduced to igt and respected.

> Do we want somehow enforce on adding docs?

Yep, we will have a requirement for all the new tests to be described
(if it makes sense).

> How deep in description we should be? Like - this test is 'doing
> reset' or is checking HW/SW feature?

Very "shallow". As I have said - I don't want C translated to english.
I want to have the intention of test captured.

Let me give you an rough example:

####################### kms_hdmi_inject.c #############################

igt_describe_test("Test that in-kernel EDID parsing is producing "
		  "expected results by forcing a HDMI connector with "
		  "a known EDID and checking that the metadata exposed "
		  "to user space matches.");
igt_main
{
	igt_describe("Make sure that 4K modes exposed by DRM match the
		     "forced EDID and the modesetting using it works.");
	igt_subtest("inject-4k")
		hdmi_inject_4k(drm_fd, connector);

	igt_describe("Make sure that audio information exposed by ALSA "
	             "match the forced EDID.");
	igt_subtest("inject-audio")
		hdmi_inject_audio(drm_fd, connector);
}

#######################################################################

This is already on the verge of translation, but that's because the test
are extremely simple. If we would expand the test a bit (let's say to
test more than a single EDID or do more detailed checks), the
description would stay true and would not require changing.

Another example, for a much more complex test could be something like this:

igt@kms_cursor_legacy@cursor-vs-flip-*
"Check that the asynchronous legacy cursor updates do not stall for a
page-flip, by making sure that we can squeeze many cursor updates
in-between flips."

The final series will come with some documentation on how to write good
descriptions (with list of dos and don'ts) and few a examples.

We can hold a biksheeding festival there :-)

-- 
Cheers,
Arek
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [RFC] IGT Subtest Documentation

[Arkadiusz Hiler]

On Mon, Jun 03, 2019 at 06:09:13PM +0300, Kalamarz, Lukasz wrote:
> LGTM:) Had one question. With test description that will include
> reasoning for a given testcase/binary, should it be duplicated in
> commit message?

It really depends. Meaningful commit messages help the reviewers to
grasp the idea of the change and are qutie helpful in any future
archeological excercises.

You won't have to copy the full thing, but giving some context for the
change is always appreciated.

-- 
Cheers,
Arek
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [RFC] IGT Subtest Documentation

[Katarzyna Dec]

On Wed, Jun 05, 2019 at 01:06:42PM +0300, Arkadiusz Hiler wrote:
> On Mon, Jun 03, 2019 at 04:51:46PM +0200, Katarzyna Dec wrote:
> > Wow Arek! That sound awsome. I hope that something in that shape will be
> > introduced to igt and respected.
> 
> > Do we want somehow enforce on adding docs?
> 
> Yep, we will have a requirement for all the new tests to be described
> (if it makes sense).
> 
> > How deep in description we should be? Like - this test is 'doing
> > reset' or is checking HW/SW feature?
> 
> Very "shallow". As I have said - I don't want C translated to english.
> I want to have the intention of test captured.
> 
> Let me give you an rough example:
> 
> ####################### kms_hdmi_inject.c #############################
> 
> igt_describe_test("Test that in-kernel EDID parsing is producing "
> 		  "expected results by forcing a HDMI connector with "
> 		  "a known EDID and checking that the metadata exposed "
> 		  "to user space matches.");
> igt_main
> {
> 	igt_describe("Make sure that 4K modes exposed by DRM match the
> 		     "forced EDID and the modesetting using it works.");
> 	igt_subtest("inject-4k")
> 		hdmi_inject_4k(drm_fd, connector);
> 
> 	igt_describe("Make sure that audio information exposed by ALSA "
> 	             "match the forced EDID.");
> 	igt_subtest("inject-audio")
> 		hdmi_inject_audio(drm_fd, connector);
> }
> 
> #######################################################################
> 
> This is already on the verge of translation, but that's because the test
> are extremely simple. If we would expand the test a bit (let's say to
> test more than a single EDID or do more detailed checks), the
> description would stay true and would not require changing.
> 
> Another example, for a much more complex test could be something like this:
> 
> igt@kms_cursor_legacy@cursor-vs-flip-*
> "Check that the asynchronous legacy cursor updates do not stall for a
> page-flip, by making sure that we can squeeze many cursor updates
> in-between flips."
> 
> The final series will come with some documentation on how to write good
> descriptions (with list of dos and don'ts) and few a examples.
> 
> We can hold a biksheeding festival there :-)
> 
> -- 
> Cheers,
> Arek
Good explanation, thanks for that.
So when do we start? :)
Kasia :)
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev