Re: [igt-dev] [PATCH i-g-t 5/5] docs: Embed subtest descriptions in the documentation

["Ser, Simon" <simon.ser@intel.com>]

On Mon, 2019-06-17 at 13:54 +0300, Arkadiusz Hiler wrote:
> This rewrites generate_description_xml in Python, so that we generate
> properly escaped XML. The switch also makes the code more manageable.
> 
> Changes in the generated docbook:
> 
> 1. subtests are not simply listed anymore, they are now another (sub)section
> 
> 2. subtests are now linkable,
>    e.g. docs/igt-kms-tests.html#kms_hdmi_inject@inject-4k
> 
> 3. subtest's section now includes output of --describe
> 
> Python is required already by gtk-doc and we are not using anything
> other than the standard library.
> 
> Cc: Daniel Vetter <daniel@ffwll.ch>
> Cc: Petri Latvala <petri.latvala@intel.com>
> Signed-off-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
> ---
>  .../igt-gpu-tools/generate_description_xml.py | 112 ++++++++++++++++++
>  .../igt-gpu-tools/generate_description_xml.sh |  46 -------
>  docs/reference/igt-gpu-tools/meson.build      |   2 +-
>  3 files changed, 113 insertions(+), 47 deletions(-)
>  create mode 100755 docs/reference/igt-gpu-tools/generate_description_xml.py
>  delete mode 100644 docs/reference/igt-gpu-tools/generate_description_xml.sh
> 
> diff --git a/docs/reference/igt-gpu-tools/generate_description_xml.py b/docs/reference/igt-gpu-tools/generate_description_xml.py
> new file mode 100755
> index 00000000..87af945d
> --- /dev/null
> +++ b/docs/reference/igt-gpu-tools/generate_description_xml.py
> @@ -0,0 +1,112 @@
> +#!/usr/bin/env python3
> +
> +import re
> +import sys
> +import os.path
> +import subprocess
> +import xml.etree.cElementTree as ET
> +
> +from collections import namedtuple
> +
> +Subtest = namedtuple("Subtest", "name description")

I'm not sure I understand "name description". Shouldn't it just be
"Description"?

> +KEYWORDS=re.compile(r'\b(invalid|hang|swap|thrash|crc|tiled|tiling|rte|ctx|render|blt|bsd|vebox|exec|rpm)\b')
> +
> +
> +def get_testlist(path):
> +    "read binaries' names from test-list.txt"
> +    with open(path, 'r') as f:
> +        assert(f.readline() == "TESTLIST\n")
> +        tests = f.readline().strip().split(" ")
> +        assert(f.readline() == "END TESTLIST\n")
> +
> +    return tests
> +
> +
> +def keywordize(root, text, keywords):
> +    "set text for root element and wrap KEYWORDS in a <acronym>"
> +    matches = list(keywords.finditer(text))
> +
> +    if not matches:
> +        root.text = text
> +        return
> +
> +    pos = 0
> +    last_element = None
> +    root.text = ""
> +
> +    for match in matches:
> +        if match.start() > pos:
> +            to_append = text[pos:match.start()]
> +
> +            if last_element == None:
> +                root.text += to_append
> +            else:
> +                last_element.tail += to_append
> +
> +        last_element = ET.SubElement(root, "acronym")
> +        last_element.tail = ""
> +        last_element.text=match.group()
> +        pos = match.end()

This misses the text after the last match.

> +def get_subtests(testdir, test):
> +    "execute test and get subtests with their descriptions via --describe"
> +    output = []
> +    full_test_path = os.path.join(testdir, test)
> +    proc = subprocess.run([full_test_path, "--describe"], stdout=subprocess.PIPE)

Add check=True to make sure the exit status is 0.

> +    description = ""
> +    current_subtest = None
> +
> +    for line in proc.stdout.decode().splitlines():
> +        if line.startswith("SUB "):
> +            output += [Subtest(current_subtest, description)]

Should only do that if description != "".

> +            description = ""
> +            current_subtest = line.split(' ')[1]
> +        else:
> +            description += line
> +
> +    output += [Subtest(current_subtest, description)]

Ditto, although this one is more like a nit.

> +    return output
> +
> +def main():
> +    output_file   = sys.argv[1]
> +    test_filter   = re.compile(sys.argv[2])
> +    testlist_file = sys.argv[3]
> +    testdir       = os.path.abspath(os.path.dirname(testlist_file))
> +
> +    root = ET.Element("refsect1")
> +    ET.SubElement(root, "title").text = "Description"
> +
> +    tests = get_testlist(testlist_file)
> +
> +    for test in tests:
> +        if not test_filter.match(test):
> +            continue
> +
> +        test_section = ET.SubElement(root, "refsect2", id=test)
> +        test_title = ET.SubElement(test_section, "title")
> +        keywordize(test_title, test, KEYWORDS)
> +
> +        subtests = get_subtests(testdir, test)
> +
> +        # we have description with no subtest name, add it at the top level
> +        if subtests and not subtests[0].name:
> +            ET.SubElement(test_section, "para").text = subtests[0].description
> +
> +        if len(subtests) > 100:
> +            ET.SubElement(test_section, "para").text = "More than 100 subtests, skipping listing"
> +            continue
> +
> +        for name, description in subtests:
> +            if not name:
> +                continue
> +
> +            subtest_section = ET.SubElement(test_section, "refsect3", id="{}@{}".format(test, name))
> +            subtest_title = ET.SubElement(subtest_section, "title")
> +            keywordize(subtest_title, name, KEYWORDS)
> +            ET.SubElement(subtest_section, "para").text = description
> +
> +    ET.ElementTree(root).write(output_file)
> +
> +main()
> diff --git a/docs/reference/igt-gpu-tools/generate_description_xml.sh b/docs/reference/igt-gpu-tools/generate_description_xml.sh
> deleted file mode 100644
> index 705a7bf3..00000000
> --- a/docs/reference/igt-gpu-tools/generate_description_xml.sh
> +++ /dev/null
> @@ -1,46 +0,0 @@
> -#!/bin/sh
> -
> -output=$1
> -filter=$2
> -testlist=$3
> -testdir=$(dirname $testlist)
> -
> -KEYWORDS="(invalid|hang|swap|thrash|crc|tiled|tiling|rte|ctx|render|blt|bsd|vebox|exec|rpm)"
> -
> -echo "<?xml version=\"1.0\"?>" > $output
> -echo "<!DOCTYPE refsect1 PUBLIC \"-//OASIS//DTD DocBook XML V4.3//EN\"" >> $output
> -echo "               \"http://www.oasis-open.org/docbook/xml/4.3/docbookx.dtd\"" >> $output
> -echo "[" >> $output
> -echo "  <!ENTITY % local.common.attrib \"xmlns:xi  CDATA  #FIXED 'http://www.w3.org/2003/XInclude'\">;" >> $output
> -echo "  <!ENTITY version SYSTEM \"version.xml\">" >> $output
> -echo "]>" >> $output
> -echo "<refsect1>" >> $output
> -echo "<title>Description</title>" >> $output
> -for test in `cat $testlist | tr ' ' '\n' | grep "^$filter" | sort`; do
> -	echo "<refsect2 id=\"$test\"><title>" >> $output;
> -	echo "$test" | perl -pe "s/(?<=_)$KEYWORDS(?=(_|\\W))/<acronym>\\1<\\/acronym>/g" >> $output;
> -	echo "</title><para><![CDATA[" >> $output;
> -	testprog=$testdir/$test;
> -	 ./$testprog --help-description >> $output;
> -	echo "]]></para>" >> $output;
> -	if ./$testprog --list-subtests > /dev/null ; then
> -		echo "<refsect3><title>Subtests</title>" >> $output;
> -		subtest_list=`./$testprog --list-subtests`;
> -		subtest_count=`echo $subtest_list | wc -w`;
> -		if [ $subtest_count -gt 100 ]; then
> -			echo "<para>This test has over 100 subtests. " >> $output;
> -			echo "Run <command>$test</command> <option>--list-subtests</option> to list them.</para>" >> $output;
> -		else
> -			echo "<simplelist>" >> $output;
> -			for subtest in $subtest_list; do
> -				echo "<member>" >> $output;
> -				echo "$subtest" | perl -pe "s/\\b$KEYWORDS\\b/<acronym>\\1<\\/acronym>/g" >> $output;
> -				echo "</member>" >> $output;
> -			done;
> -			echo "</simplelist>" >> $output;
> -		fi;
> -		echo "</refsect3>" >> $output;
> -	fi;
> -	echo "</refsect2>" >> $output;
> -done;
> -echo "</refsect1>" >> $output
> diff --git a/docs/reference/igt-gpu-tools/meson.build b/docs/reference/igt-gpu-tools/meson.build
> index 4d177e49..e2bdc495 100644
> --- a/docs/reference/igt-gpu-tools/meson.build
> +++ b/docs/reference/igt-gpu-tools/meson.build
> @@ -45,7 +45,7 @@ test_groups = [
>  	'vgem',
>  ]
>  
> -gen_description = find_program('generate_description_xml.sh')
> +gen_description = find_program('generate_description_xml.py')
>  gen_programs = find_program('generate_programs_xml.sh')
>  
>  generated_docs = []
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev