[LTP] [PATCH v5 0/2] Tests catalog

[LTP] [PATCH v5 0/2] Tests catalog

[Andrea Cervesato]

This patch-set is meant to introduce a new page in the LTP
documentation, showing tests which are currently available with their
description and information.

Signed-off-by: Andrea Cervesato <andrea.cervesato@suse.com>
---
Changes in v5:
- add hugepages, ulimit, resource_files, save_restore
- add more comments in the code
- fix multiline table cells
- Link to v4: https://lore.kernel.org/r/20250205-doc_tests_list-v4-1-ae46e56073bb@suse.com

Changes in v4:
- show maximum runtime with timeout text
- correct timeout text
- Link to v3: https://lore.kernel.org/r/20250205-doc_tests_list-v3-1-b60649cdea6a@suse.com

Changes in v3:
- fix table widths alignment
- add warning at the beginning of the test catalog
- Link to v2: https://lore.kernel.org/r/20250204-doc_tests_list-v2-1-9d29aeacc3f3@suse.com

Changes in v2:
- add setup
- cleanup code
- Link to v1: https://lore.kernel.org/r/20250203-doc_tests_list-v1-1-5fe69c07b1ab@suse.com

---
Andrea Cervesato (2):
      doc: remove margin for multiline cells
      doc: add tests catalog page

 doc/.gitignore             |   1 +
 doc/_static/custom.css     |   5 +
 doc/conf.py                | 274 ++++++++++++++++++++++++++++++++++++++++++++-
 doc/index.rst              |   4 +
 doc/users/test_catalog.rst |   7 ++
 5 files changed, 287 insertions(+), 4 deletions(-)
---
base-commit: 7f3922a2499ebff6134c37a17a0de6012a318fb4
change-id: 20250131-doc_tests_list-1b82f51e43fd

Best regards,
-- 
Andrea Cervesato <andrea.cervesato@suse.com>


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v5 1/2] doc: remove margin for multiline cells

[Andrea Cervesato]

From: Andrea Cervesato <andrea.cervesato@suse.com>

Multiline table's cells show a margin at the end of them. Remove the
margin by setting 'margin-bottom: 0' in the CSS table declaration
style.

Signed-off-by: Andrea Cervesato <andrea.cervesato@suse.com>
---
 doc/_static/custom.css | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/doc/_static/custom.css b/doc/_static/custom.css
index cb4d3821dc65f9f904cd90951f968bc90d8798c1..c5816a55346fa28416377375bc91966780075dc3 100644
--- a/doc/_static/custom.css
+++ b/doc/_static/custom.css
@@ -2,3 +2,8 @@
 .wy-table-responsive table td {
     white-space: normal;
 }
+
+/* remove margin for multiline cells */
+.rst-content table td div.line-block {
+    margin-bottom: 0;
+}

-- 
2.43.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v5 2/2] doc: add tests catalog page

[Andrea Cervesato]

From: Andrea Cervesato <andrea.cervesato@suse.com>

Add a new section in the LTP documentation website, where we list all
tests which are available in LTP and supporting new API.

Acked-by: Cyril Hrubis <chrubis@suse.cz>
Signed-off-by: Andrea Cervesato <andrea.cervesato@suse.com>
---
 doc/.gitignore             |   1 +
 doc/conf.py                | 274 ++++++++++++++++++++++++++++++++++++++++++++-
 doc/index.rst              |   4 +
 doc/users/test_catalog.rst |   7 ++
 4 files changed, 282 insertions(+), 4 deletions(-)

diff --git a/doc/.gitignore b/doc/.gitignore
index 173179852070f25acb6a729975df9d52d171b422..2b05a1ec368573778cfe7ee6a1cb5d6c5ecb0b5a 100644
--- a/doc/.gitignore
+++ b/doc/.gitignore
@@ -1,4 +1,5 @@
 html/
 build/
 _static/syscalls.rst
+_static/tests.rst
 syscalls.tbl
diff --git a/doc/conf.py b/doc/conf.py
index c6a84ea5810424ce6e1c21d81946c1819f10a3cc..0bebd0e2426c1c4155e39c1db918f38d92e88fd8 100644
--- a/doc/conf.py
+++ b/doc/conf.py
@@ -5,6 +5,7 @@
 
 import os
 import re
+import json
 import socket
 import urllib.request
 import sphinx
@@ -17,6 +18,7 @@ copyright = '2024, Linux Test Project'
 author = 'Linux Test Project'
 release = '1.0'
 ltp_repo = 'https://github.com/linux-test-project/ltp'
+ltp_repo_base_url = f"{ltp_repo}/tree/master"
 
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
@@ -25,7 +27,7 @@ extensions = [
     'linuxdoc.rstKernelDoc',
     'sphinxcontrib.spelling',
     'sphinx.ext.autosectionlabel',
-    'sphinx.ext.extlinks'
+    'sphinx.ext.extlinks',
 ]
 
 exclude_patterns = ["html*", '_static*']
@@ -138,7 +140,6 @@ def generate_syscalls_stats(_):
     if error:
         return
 
-    syscalls_base_url = f"{ltp_repo}/tree/master"
     text = [
         'Syscalls\n',
         '--------\n\n',
@@ -176,7 +177,7 @@ def generate_syscalls_stats(_):
             path = dirpath.replace('../', '')
             name = match.group('name')
 
-            ltp_syscalls[name] = f'{syscalls_base_url}/{path}'
+            ltp_syscalls[name] = f'{ltp_repo_base_url}/{path}'
 
     # compare kernel syscalls with LTP tested syscalls
     syscalls = {}
@@ -186,7 +187,7 @@ def generate_syscalls_stats(_):
 
         if kersc not in syscalls:
             if kersc in white_list:
-                syscalls[kersc] = f'{syscalls_base_url}/{white_list[kersc]}'
+                syscalls[kersc] = f'{ltp_repo_base_url}/{white_list[kersc]}'
                 continue
 
             syscalls[kersc] = None
@@ -256,6 +257,270 @@ def generate_syscalls_stats(_):
         stats.writelines(text)
 
 
+def _generate_tags_table(tags):
+    """
+    Generate the tags table from tags hash.
+    """
+    supported_url_ref = {
+        "linux-git": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=",
+        "linux-stable-git": "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=",
+        "glibc-git": "https://sourceware.org/git/?p=glibc.git;a=commit;h=",
+        "musl-git": "https://git.musl-libc.org/cgit/musl/commit/src/linux/clone.c?id=",
+        "CVE": "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-",
+    }
+
+    table = [
+        '.. list-table::',
+        '   :header-rows: 1',
+        '',
+        '   * - Tag',
+        '     - Info',
+    ]
+
+    for tag in tags:
+        tag_key = tag[0]
+        tag_val = tag[1]
+
+        tag_url = supported_url_ref.get(tag_key, None)
+        if tag_url:
+            tag_val = f'`{tag_val} <{tag_url}{tag_val}>`_'
+
+        table.extend([
+            f'   * - {tag_key}',
+            f'     - {tag_val}',
+        ])
+
+    return table
+
+
+def _generate_options_table(options):
+    """
+    Generate the options table from the options hash.
+    """
+    table = [
+        '.. list-table::',
+        '   :header-rows: 1',
+        '',
+        '   * - Option',
+        '     - Description',
+    ]
+
+    for opt in options:
+        if not isinstance(opt, list):
+            table.clear()
+            break
+
+        key = opt[0]
+        val = opt[2]
+
+        if key.endswith(':'):
+            key = key[:-1] if key.endswith(':') else key
+
+        key = f'-{key}'
+
+        table.extend([
+            f'   * - {key}',
+            f'     - {val}',
+        ])
+
+    return table
+
+
+def _generate_table_cell(key, values):
+    """
+    Generate a cell which can be multiline if value is a list.
+    """
+    cell = []
+
+    if len(values) > 1:
+        cell.extend([
+            f'   * - {key}',
+            f'     - | {values[0]}',
+        ])
+
+        for item in values[1:]:
+            cell.append(f'       | {item}')
+    else:
+        cell.extend([
+            f'   * - {key}',
+            f'     - {values[0]}',
+        ])
+
+    return cell
+
+
+def _generate_setup_table(keys):
+    """
+    Generate the table with test setup configuration.
+    """
+    exclude = [
+        # following keys are already handled
+        'options',
+        'runtime',
+        'timeout',
+        'fname',
+        'doc',
+        # following keys don't need to be shown
+        'child_needs_reinit',
+        'needs_checkpoints',
+        'forks_child',
+        'tags',
+    ]
+    my_keys = {k: v for k, v in keys.items() if k not in exclude}
+    if len(my_keys) == 0:
+        return []
+
+    table = [
+        '.. list-table::',
+        '   :header-rows: 1',
+        '',
+        '   * - Key',
+        '     - Value',
+    ]
+
+    values = []
+
+    for key, value in my_keys.items():
+        if key in exclude:
+            continue
+
+        values.clear()
+
+        if key == 'ulimit':
+            for item in value:
+                values.append(f'{item[0]} : {item[1]}')
+        elif key == 'hugepages':
+            if len(value) == 1:
+                values.append(f'{value[0]}')
+            else:
+                values.append(f'{value[0]}, {value[1]}')
+        elif key == 'filesystems':
+            for v in value:
+                for item in v:
+                    if isinstance(item, list):
+                        continue
+
+                    if item.startswith('.type'):
+                        values.append(item.replace('.type=', ''))
+        elif key == "save_restore":
+            for item in value:
+                values.append(item[0])
+        else:
+            if isinstance(value, list):
+                values.extend(value)
+            else:
+                values.append(value)
+
+        table.extend(_generate_table_cell(key, values))
+
+    return table
+
+
+def generate_test_catalog(_):
+    """
+    Generate the test catalog from ltp.json metadata file.
+    """
+    output = '_static/tests.rst'
+    metadata_file = '../metadata/ltp.json'
+    text = [
+        '.. warning::',
+        '    The following catalog has been generated using LTP metadata',
+        '    which is including only tests using the new :ref:`LTP C API`.',
+        ''
+    ]
+
+    metadata = None
+    with open(metadata_file, 'r', encoding='utf-8') as data:
+        metadata = json.load(data)
+
+    timeout_def = metadata['defaults']['timeout']
+
+    for test_name, conf in metadata['tests'].items():
+        text.extend([
+            f'{test_name}',
+            len(test_name) * '-'
+        ])
+
+        # source url location
+        test_fname = conf.get('fname', None)
+        if test_fname:
+            text.extend([
+                '',
+                f"`source <{ltp_repo_base_url}/{test_fname}>`__",
+                ''
+            ])
+
+        # test description
+        desc = conf.get('doc', None)
+        if desc:
+            desc_text = []
+            for line in desc:
+                if line.startswith("[Description]"):
+                    desc_text.append("**Description**")
+                elif line.startswith("[Algorithm]"):
+                    desc_text.append("**Algorithm**")
+                else:
+                    desc_text.append(line)
+
+            text.extend([
+                '\n'.join(desc_text),
+            ])
+
+        # timeout information
+        timeout = conf.get('timeout', None)
+        if timeout:
+            text.extend([
+                '',
+                f'Test timeout is {timeout} seconds.',
+            ])
+        else:
+            text.extend([
+                '',
+                f'Test timeout defaults is {timeout_def} seconds.',
+            ])
+
+        # runtime information
+        runtime = conf.get('runtime', None)
+        if runtime:
+            text.extend([
+                f'Maximum runtime is {runtime} seconds.',
+                ''
+            ])
+        else:
+            text.append('')
+
+        # options information
+        opts = conf.get('options', None)
+        if opts:
+            text.append('')
+            text.extend(_generate_options_table(opts))
+            text.append('')
+
+        # tags information
+        tags = conf.get('tags', None)
+        if tags:
+            text.append('')
+            text.extend(_generate_tags_table(tags))
+            text.append('')
+
+        # parse struct tst_test content
+        text.append('')
+        text.extend(_generate_setup_table(conf))
+        text.append('')
+
+        # small separator between tests
+        text.extend([
+            '',
+            '.. raw:: html',
+            '',
+            '    <hr>',
+            '',
+        ])
+
+    with open(output, 'w+', encoding='utf-8') as new_tests:
+        new_tests.write('\n'.join(text))
+
+
 def setup(app):
     """
     Setup the current documentation, using self generated data and graphics
@@ -263,3 +528,4 @@ def setup(app):
     """
     app.add_css_file('custom.css')
     app.connect('builder-inited', generate_syscalls_stats)
+    app.connect('builder-inited', generate_test_catalog)
diff --git a/doc/index.rst b/doc/index.rst
index b907ac36f0c9328c576d25dee5777d808c2e5119..c00a59d31345142e78deb74eacc9da2941291d76 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -11,6 +11,7 @@
    users/setup_tests
    users/supported_systems
    users/stats
+   users/test_catalog
 
 .. toctree::
    :maxdepth: 3
@@ -54,6 +55,9 @@ For users
 :doc:`users/stats`
    Some LTP statistics
 
+:doc:`users/test_catalog`
+   The LTP test catalog
+
 For developers
 --------------
 
diff --git a/doc/users/test_catalog.rst b/doc/users/test_catalog.rst
new file mode 100644
index 0000000000000000000000000000000000000000..b1674f9dc614ea04a89cf084e92b72c6862a5f48
--- /dev/null
+++ b/doc/users/test_catalog.rst
@@ -0,0 +1,7 @@
+.. SPDX-License-Identifier: GPL-2.0-or-later
+
+Test catalog
+============
+
+.. include:: ../_static/tests.rst
+

-- 
2.43.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v5 1/2] doc: remove margin for multiline cells

[Petr Vorel]

Hi Andrea,

Suggested-by: Petr Vorel <pvorel@suse.cz>
Reviewed-by: Petr Vorel <pvorel@suse.cz>

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v5 2/2] doc: add tests catalog page

[Petr Vorel]

Hi Andrea,

Acked-by: Petr Vorel <pvorel@suse.cz>

Thanks for extra comments to _generate_setup_table().

There is quite a lot of warnings. I'm not suggesting we should fix them (I don't
want to block this, let's merge it today), but more warnings and errors we get
the worse is to find a real problem which might appear in the future.

I might take time in the future to fix them - some of them could be easily
fixed and some of the warning means part of the description got lost, e.g.
"1. First test case" and "2. Second test case" in mountns0[1-3].c is not
followed by the actual description "X: bind mounts DIR_B to DIR_A", indent is
also wrong. This means sphinx syntax (reST) is not 100% compatible to
asciidoc{,tor} syntax. Syntax problems are also visible in Vim with highlighting
on.

_static/tests.rst:557: WARNING: Inline interpreted text or phrase reference start-string without end-string.

The code `mmap(2)`s reg
=> using unnecessary `...` (for formatting as code it would be ``..``, but I
would just remove it.

_static/tests.rst:693: WARNING: Bullet list ends without a blank line; unexpected unindent.
_static/tests.rst:694: WARNING: Enumerated list ends without a blank line; unexpected unindent.
_static/tests.rst:697: WARNING: Block quote ends without a blank line; unexpected unindent.
_static/tests.rst:698: WARNING: Enumerated list ends without a blank line; unexpected unindent.

_static/tests.rst:742: WARNING: Bullet list ends without a blank line; unexpected unindent.
_static/tests.rst:743: WARNING: Enumerated list ends without a blank line; unexpected unindent.
_static/tests.rst:746: WARNING: Block quote ends without a blank line; unexpected unindent.
_static/tests.rst:747: WARNING: Enumerated list ends without a blank line; unexpected unindent.
_static/tests.rst:792: WARNING: Bullet list ends without a blank line; unexpected unindent.
_static/tests.rst:793: WARNING: Enumerated list ends without a blank line; unexpected unindent.

These are the lost description.

_static/tests.rst:796: WARNING: Block quote ends without a blank line; unexpected unindent.
_static/tests.rst:797: WARNING: Enumerated list ends without a blank line; unexpected unindent.

_static/tests.rst:2530: ERROR: Unexpected indentation.

=> FYI  this one might be '.' instead of '*'

. It writes instead of reads to fill the page cache. Because no
  pages were allocated on tmpfs.

_static/tests.rst:2531: WARNING: Block quote ends without a blank line; unexpected unindent.
_static/tests.rst:2533: ERROR: Unexpected indentation.
_static/tests.rst:9425: WARNING: Inline emphasis start-string without end-string.
_static/tests.rst:11685: WARNING: Bullet list ends without a blank line; unexpected unindent.
_static/tests.rst:12122: WARNING: Bullet list ends without a blank line; unexpected unindent.
_static/tests.rst:14986: ERROR: Unexpected indentation.
_static/tests.rst:14987: WARNING: Block quote ends without a blank line; unexpected unindent.
_static/tests.rst:15779: ERROR: Unexpected indentation.
_static/tests.rst:15781: WARNING: Block quote ends without a blank line; unexpected unindent.
_static/tests.rst:15787: WARNING: Definition list ends without a blank line; unexpected unindent.
_static/tests.rst:17886: WARNING: Enumerated list ends without a blank line; unexpected unindent.
_static/tests.rst:17887: WARNING: Block quote ends without a blank line; unexpected unindent.
_static/tests.rst:17888: WARNING: Enumerated list ends without a blank line; unexpected unindent.
_static/tests.rst:24631: ERROR: Unexpected indentation.
_static/tests.rst:24632: WARNING: Block quote ends without a blank line; unexpected unindent.
_static/tests.rst:24766: ERROR: Unexpected indentation.
_static/tests.rst:27685: WARNING: Enumerated list ends without a blank line; unexpected unindent.
_static/tests.rst:27820: WARNING: Bullet list ends without a blank line; unexpected unindent.
_static/tests.rst:33084: WARNING: Bullet list ends without a blank line; unexpected unindent.
_static/tests.rst:33085: WARNING: Block quote ends without a blank line; unexpected unindent.
_static/tests.rst:33610: WARNING: Bullet list ends without a blank line; unexpected unindent.
_static/tests.rst:34692: ERROR: Unexpected indentation.
_static/tests.rst:34694: WARNING: Block quote ends without a blank line; unexpected unindent.
_static/tests.rst:34722: ERROR: Unexpected indentation.
_static/tests.rst:35279: WARNING: Bullet list ends without a blank line; unexpected unindent.
_static/tests.rst:35309: WARNING: Bullet list ends without a blank line; unexpected unindent.
_static/tests.rst:39506: WARNING: Bullet list ends without a blank line; unexpected unindent.
_static/tests.rst:41852: WARNING: Inline emphasis start-string without end-string.
_static/tests.rst:41855: WARNING: Inline emphasis start-string without end-string.
_static/tests.rst:41858: WARNING: Inline emphasis start-string without end-string.
_static/tests.rst:41862: WARNING: Inline emphasis start-string without end-string.
_static/tests.rst:41866: WARNING: Inline emphasis start-string without end-string.
_static/tests.rst:41869: WARNING: Inline emphasis start-string without end-string.

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v5 0/2] Tests catalog

[Andrea Cervesato via ltp]

Hi!

Merged, thanks for the reviews.

Andrea

On 2/6/25 11:16, Andrea Cervesato wrote:
> This patch-set is meant to introduce a new page in the LTP
> documentation, showing tests which are currently available with their
> description and information.
>
> Signed-off-by: Andrea Cervesato <andrea.cervesato@suse.com>
> ---
> Changes in v5:
> - add hugepages, ulimit, resource_files, save_restore
> - add more comments in the code
> - fix multiline table cells
> - Link to v4: https://lore.kernel.org/r/20250205-doc_tests_list-v4-1-ae46e56073bb@suse.com
>
> Changes in v4:
> - show maximum runtime with timeout text
> - correct timeout text
> - Link to v3: https://lore.kernel.org/r/20250205-doc_tests_list-v3-1-b60649cdea6a@suse.com
>
> Changes in v3:
> - fix table widths alignment
> - add warning at the beginning of the test catalog
> - Link to v2: https://lore.kernel.org/r/20250204-doc_tests_list-v2-1-9d29aeacc3f3@suse.com
>
> Changes in v2:
> - add setup
> - cleanup code
> - Link to v1: https://lore.kernel.org/r/20250203-doc_tests_list-v1-1-5fe69c07b1ab@suse.com
>
> ---
> Andrea Cervesato (2):
>        doc: remove margin for multiline cells
>        doc: add tests catalog page
>
>   doc/.gitignore             |   1 +
>   doc/_static/custom.css     |   5 +
>   doc/conf.py                | 274 ++++++++++++++++++++++++++++++++++++++++++++-
>   doc/index.rst              |   4 +
>   doc/users/test_catalog.rst |   7 ++
>   5 files changed, 287 insertions(+), 4 deletions(-)
> ---
> base-commit: 7f3922a2499ebff6134c37a17a0de6012a318fb4
> change-id: 20250131-doc_tests_list-1b82f51e43fd
>
> Best regards,

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v5 2/2] doc: add tests catalog page

[Andrea Cervesato via ltp]

Hi Petr,

On 2/6/25 15:26, Petr Vorel wrote:
> Hi Andrea,
>
> Acked-by: Petr Vorel<pvorel@suse.cz>
>
> Thanks for extra comments to _generate_setup_table().
>
> There is quite a lot of warnings. I'm not suggesting we should fix them (I don't
> want to block this, let's merge it today), but more warnings and errors we get
> the worse is to find a real problem which might appear in the future.
>
> I might take time in the future to fix them - some of them could be easily
> fixed and some of the warning means part of the description got lost, e.g.
> "1. First test case" and "2. Second test case" in mountns0[1-3].c is not
> followed by the actual description "X: bind mounts DIR_B to DIR_A", indent is
> also wrong. This means sphinx syntax (reST) is not 100% compatible to
> asciidoc{,tor} syntax. Syntax problems are also visible in Vim with highlighting
> on.
The warnings are all related to the asciidoc format which needs to be 
converted into RST.
The meaning of these patches was also to introduce this change.

Andrea


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v5 2/2] doc: add tests catalog page

[Petr Vorel]

> Hi Petr,

> On 2/6/25 15:26, Petr Vorel wrote:
> > Hi Andrea,

> > Acked-by: Petr Vorel<pvorel@suse.cz>

> > Thanks for extra comments to _generate_setup_table().

> > There is quite a lot of warnings. I'm not suggesting we should fix them (I don't
> > want to block this, let's merge it today), but more warnings and errors we get
> > the worse is to find a real problem which might appear in the future.

> > I might take time in the future to fix them - some of them could be easily
> > fixed and some of the warning means part of the description got lost, e.g.
> > "1. First test case" and "2. Second test case" in mountns0[1-3].c is not
> > followed by the actual description "X: bind mounts DIR_B to DIR_A", indent is
> > also wrong. This means sphinx syntax (reST) is not 100% compatible to
> > asciidoc{,tor} syntax. Syntax problems are also visible in Vim with highlighting
> > on.
> The warnings are all related to the asciidoc format which needs to be
> converted into RST.

OK. I'll have look into the fixes tomorrow.
I'll also prepare the docparse/ removal.

> The meaning of these patches was also to introduce this change.

Ah, I was not aware of the slight difference until seeing it in the action.
But sure, not a big deal.

Kind regards,
Petr

> Andrea


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp