Re: [PATCH v3] Documentation: fix build with Asciidoctor 2

["SZEDER Gábor" <szeder.dev@gmail.com>]

On Sat, Sep 14, 2019 at 07:49:19PM +0000, brian m. carlson wrote:
> Our documentation toolchain has traditionally been built around DocBook
> 4.5.  This version of DocBook is the last DTD-based version of DocBook.
> In 2009, DocBook 5 was introduced using namespaces and its syntax is
> expressed in RELAX NG, which is more expressive and allows a wider
> variety of syntax forms.
> 
> Asciidoctor, one of the alternatives for building our documentation,
> moved support for DocBook 4.5 out of core in its recent 2.0 release and
> now only supports DocBook 5 in the main release.  The DocBoook 4.5
> converter is still available as a separate component, but this is not
> available in most distro packages.  This would not be a problem but for
> the fact that we use xmlto, which is still stuck in the DocBook 4.5 era.
> 
> xmlto performs DTD validation as part of the build process.  This is not
> problematic for DocBook 4.5, which has a valid DTD, but it clearly
> cannot work for DocBook 5, since no DTD can adequately express its full
> syntax.  In addition, even if xmlto did support RELAX NG validation,
> that wouldn't be sufficient because it uses the libxml2-based xmllint to
> do so, which has known problems with validating interleaves in RELAX NG.
> 
> Fortunately, there's an easy way forward: ask Asciidoctor to use its
> DocBook 5 backend and tell xmlto to skip validation.  Asciidoctor has
> supported DocBook 5 since v0.1.4 in 2013 and xmlto has supported
> skipping validation for probably longer than that.
> 
> We also need to teach xmlto how to use the namespaced DocBook XSLT
> stylesheets instead of the non-namespaced ones it usually uses.
> Normally these stylesheets are interchangeable, but the non-namespaced
> ones have a bug that causes them not to strip whitespace automatically
> from certain elements when namespaces are in use.  This results in
> additional whitespace at the beginning of list elements, which is
> jarring and unsightly.
> 
> We can do this by passing a custom stylesheet with the -x option that
> simply imports the namespaced stylesheets via a URL.  Any system with
> support for XML catalogs will automatically look this URL up and
> reference a local copy instead without us having to know where this
> local copy is located.  We know that anyone using xmlto will already
> have catalogs set up properly since the DocBook 4.5 DTD used during
> validation is also looked up via catalogs.  All major Linux
> distributions distribute the necessary stylesheets and have built-in
> catalog support, and Homebrew does as well, albeit with a requirement to
> set an environment variable to enable catalog support.
> 
> On the off chance that someone lacks support for catalogs, it is
> possible for xmlto (via xmllint) to download the stylesheets from the
> URLs in question, although this will likely perform poorly enough to
> attract attention.  People still have the option of using the prebuilt
> documentation that we ship, so happily this should not be an impediment.
> 
> Finally, we need to filter out some messages from other stylesheets that
> occur when invoking dblatex in the CI job.  This tool strips namespaces
> much like the unnamespaced DocBook stylesheets and prints similar
> messages.  If we permit these messages to be printed to standard error,
> our documentation CI job will fail because we check standard error for
> unexpected output.  Due to dblatex's reliance on Python 2, we may need
> to revisit its use in the future, in which case this problem may go
> away, but this can be delayed until a future patch.
> 
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>

>  Documentation/Makefile     | 4 +++-
>  Documentation/manpage.xsl  | 3 +++
>  azure-pipelines.yml        | 2 +-
>  ci/install-dependencies.sh | 2 +-
>  ci/test-documentation.sh   | 2 ++
>  5 files changed, 10 insertions(+), 3 deletions(-)
>  create mode 100644 Documentation/manpage.xsl
> 
> diff --git a/Documentation/Makefile b/Documentation/Makefile
> index 76f2ecfc1b..d94f47c5c9 100644
> --- a/Documentation/Makefile
> +++ b/Documentation/Makefile
> @@ -197,11 +197,13 @@ ifdef USE_ASCIIDOCTOR
>  ASCIIDOC = asciidoctor
>  ASCIIDOC_CONF =
>  ASCIIDOC_HTML = xhtml5
> -ASCIIDOC_DOCBOOK = docbook45
> +ASCIIDOC_DOCBOOK = docbook5
>  ASCIIDOC_EXTRA += -acompat-mode -atabsize=8
>  ASCIIDOC_EXTRA += -I. -rasciidoctor-extensions
>  ASCIIDOC_EXTRA += -alitdd='&\#x2d;&\#x2d;'
>  DBLATEX_COMMON =
> +XMLTO_EXTRA += --skip-validation
> +XMLTO_EXTRA += -x manpage.xsl
>  endif
>  
>  SHELL_PATH ?= $(SHELL)
> diff --git a/Documentation/manpage.xsl b/Documentation/manpage.xsl
> new file mode 100644
> index 0000000000..ef64bab17a
> --- /dev/null
> +++ b/Documentation/manpage.xsl
> @@ -0,0 +1,3 @@
> +<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
> +	<xsl:import href="http://docbook.sourceforge.net/release/xsl-ns/current/manpages/docbook.xsl" />
> +</xsl:stylesheet>
> diff --git a/azure-pipelines.yml b/azure-pipelines.yml
> index c329b7218b..34031b182a 100644
> --- a/azure-pipelines.yml
> +++ b/azure-pipelines.yml
> @@ -374,7 +374,7 @@ jobs:
>         test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
>  
>         sudo apt-get update &&
> -       sudo apt-get install -y asciidoc xmlto asciidoctor &&
> +       sudo apt-get install -y asciidoc xmlto asciidoctor docbook-xsl-ns &&
>  
>         export ALREADY_HAVE_ASCIIDOCTOR=yes. &&
>         export jobname=Documentation &&
> diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh
> index 8cc72503cb..a76f348484 100755
> --- a/ci/install-dependencies.sh
> +++ b/ci/install-dependencies.sh
> @@ -53,7 +53,7 @@ StaticAnalysis)
>  	;;
>  Documentation)
>  	sudo apt-get -q update
> -	sudo apt-get -q -y install asciidoc xmlto
> +	sudo apt-get -q -y install asciidoc xmlto docbook-xsl-ns

Ok, with this package installed the build passed on Travis CI.

>  	test -n "$ALREADY_HAVE_ASCIIDOCTOR" ||
>  	gem install --version 1.5.8 asciidoctor

So, since the documentation can now be built with Asciidoctor v2, is
it already time to remove this '--version 1.5.8'?

> diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh
> index d49089832d..b3e76ef863 100755
> --- a/ci/test-documentation.sh
> +++ b/ci/test-documentation.sh
> @@ -8,6 +8,8 @@
>  filter_log () {
>  	sed -e '/^GIT_VERSION = /d' \
>  	    -e '/^    \* new asciidoc flags$/d' \
> +	    -e '/stripped namespace before processing/d' \
> +	    -e '/Attributed.*IDs for element/d' \

I haven't seen this latter message in the CI builds, neither with
Asciidoctor v1.5.8 nor with v2.  Do we really need this filter, then?
Where does this message come from?

>  	    "$1"
>  }
>