[PATCH 0 of 3] improve checking for documentation tools and formatting

[PATCH 0 of 3] improve checking for documentation tools and formatting

[Matt Wilson]

This patch series is a follow up to a change that I posed for the last
Xen Documentation Day that used lynx to create plain text from
markdown files. I've added ./configure time checking for all the tools
used in the docs/ tree. The user can persistently override one of
these tools at ./configure time by setting the appropriate environment
variable. The docs/ tree maintains the list of default tools, so
running ./configure is not a prerequisite for running "make -C docs".

I've switched to using elinks instead of lynx for creating text
documentation, as it provides better formatting and interface compared
to other tools. Of course the user can override the tool and flags if
they'd like to use something else.

Here's a sample of ./configure output after this series is applied:

checking for ps2pdf... /usr/bin/ps2pdf
checking for dvips... no
configure: WARNING: dvips is not available so some documentation won't be built
checking for latex... no
configure: WARNING: latex is not available so some documentation won't be built

Please let me know if there are other concerns that need to be
addressed.

Matt

[PATCH 1 of 3] tools: check for documentation generation tools at configure time

[Matt Wilson]

It is sometimes hard to discover all the optional documentation
generation tools that should be on a system to build all available Xen
documentation. By checking for documentation generation tools at
./configure time and displaying a warning, Xen packagers will more
easily learn about new optional build dependencies, like markdown,
when they are introduced.

Signed-off-by: Matt Wilson <msw@amazon.com>

diff -r d7e4efa17fb0 -r 674b694814c8 README
--- a/README	Tue Aug 28 15:35:08 2012 -0700
+++ b/README	Wed Aug 29 11:07:52 2012 -0700
@@ -28,8 +28,10 @@
 your system. For full documentation, see the Xen User Manual. If this
 is a pre-built release then you can find the manual at:
  dist/install/usr/share/doc/xen/pdf/user.pdf
-If you have a source release, then 'make -C docs' will build the
-manual at docs/pdf/user.pdf.
+If you have a source release and the required documentation generation
+tools, then 'make -C docs' will build the manual at docs/pdf/user.pdf.
+Running ./configure will check for the full suite of documentation
+tools and will display a warning if missing tools are detected.
 
 Quick-Start Guide
 =================
@@ -59,7 +61,6 @@
     * GNU gettext
     * 16-bit x86 assembler, loader and compiler (dev86 rpm or bin86 & bcc debs)
     * ACPI ASL compiler (iasl)
-    * markdown
 
 In addition to the above there are a number of optional build
 prerequisites. Omitting these will cause the related features to be
@@ -67,6 +68,7 @@
     * Development install of Ocaml (e.g. ocaml-nox and
       ocaml-findlib). Required to build ocaml components which
       includes the alternative ocaml xenstored.
+    * markdown
 
 Second, you need to acquire a suitable kernel for use in domain 0. If
 possible you should use a kernel provided by your OS distributor. If
diff -r d7e4efa17fb0 -r 674b694814c8 config/Tools.mk.in
--- a/config/Tools.mk.in	Tue Aug 28 15:35:08 2012 -0700
+++ b/config/Tools.mk.in	Wed Aug 29 11:07:52 2012 -0700
@@ -22,6 +22,17 @@
 LD86                := @LD86@
 BCC                 := @BCC@
 IASL                := @IASL@
+PS2PDF              := @PS2PDF@
+DVIPS               := @DVIPS@
+LATEX               := @LATEX@
+FIG2DEV             := @FIG2DEV@
+LATEX2HTML          := @LATEX2HTML@
+DOXYGEN             := @DOXYGEN@
+POD2MAN             := @POD2MAN@
+POD2TEXT            := @POD2TEXT@
+DOT                 := @DOT@
+NEATO               := @NEATO@
+MARKDOWN            := @MARKDOWN@
 
 # Extra folder for libs/includes
 PREPEND_INCLUDES    := @PREPEND_INCLUDES@
diff -r d7e4efa17fb0 -r 674b694814c8 docs/Makefile
--- a/docs/Makefile	Tue Aug 28 15:35:08 2012 -0700
+++ b/docs/Makefile	Wed Aug 29 11:07:52 2012 -0700
@@ -3,6 +3,11 @@
 XEN_ROOT=$(CURDIR)/..
 include $(XEN_ROOT)/Config.mk
 include $(XEN_ROOT)/docs/Docs.mk
+# The default documentation tools specified in Docs.mk can be
+# persistently overridden by the user via ./configure, but running
+# ./configure is not required to build the docs tree. Thus Tools.mk is
+# optionally included.
+-include $(XEN_ROOT)/config/Tools.mk
 
 VERSION		= xen-unstable
 
diff -r d7e4efa17fb0 -r 674b694814c8 tools/configure.ac
--- a/tools/configure.ac	Tue Aug 28 15:35:08 2012 -0700
+++ b/tools/configure.ac	Wed Aug 29 11:07:52 2012 -0700
@@ -34,6 +34,7 @@
 m4_include([m4/curses.m4])
 m4_include([m4/pthread.m4])
 m4_include([m4/ptyfuncs.m4])
+m4_include([m4/docs_tool.m4])
 
 # Enable/disable options
 AX_ARG_DEFAULT_DISABLE([githttp], [Download GIT repositories via HTTP])
@@ -80,6 +81,17 @@
 AC_PATH_PROG([BISON], [bison])
 AC_PATH_PROG([FLEX], [flex])
 AX_PATH_PROG_OR_FAIL([PERL], [perl])
+AX_DOCS_TOOL_PROG([PS2PDF], [ps2pdf])
+AX_DOCS_TOOL_PROG([DVIPS], [dvips])
+AX_DOCS_TOOL_PROG([LATEX], [latex])
+AX_DOCS_TOOL_PROG([FIG2DEV], [fig2dev])
+AX_DOCS_TOOL_PROG([LATEX2HTML], [latex2html])
+AX_DOCS_TOOL_PROG([DOXYGEN], [doxygen])
+AX_DOCS_TOOL_PROG([POD2MAN], [pod2man])
+AX_DOCS_TOOL_PROG([POD2TEXT], [pod2text])
+AX_DOCS_TOOL_PROG([DOT], [dot])
+AX_DOCS_TOOL_PROG([NEATO], [neato])
+AX_DOCS_TOOL_PROG([MARKDOWN], [markdown])
 AS_IF([test "x$xapi" = "xy"], [
     AX_PATH_PROG_OR_FAIL([CURL], [curl-config])
     AX_PATH_PROG_OR_FAIL([XML], [xml2-config])
diff -r d7e4efa17fb0 -r 674b694814c8 tools/m4/docs_tool.m4
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/tools/m4/docs_tool.m4	Wed Aug 29 11:07:52 2012 -0700
@@ -0,0 +1,8 @@
+AC_DEFUN([AX_DOCS_TOOL_PROG], [
+dnl
+    AC_ARG_VAR([$1], [Path to $2 tool])
+    AC_PATH_PROG([$1], [$2])
+    AS_IF([! test -x "$ac_cv_path_$1"], [
+        AC_MSG_WARN([$2 is not available so some documentation won't be built])
+    ])
+])

[PATCH 2 of 3] docs: use elinks to format markdown-generated html to text

[Matt Wilson]

Markdown, while easy to read and write, isn't the most consumable
format for users reading documentation on a terminal. This patch uses
lynx to format markdown produced HTML into text files.

Changes since v3:
 * check for html to text dump tool in ./configure
 * switch to using elinks
 * allow command line flags to dump tool to be specified

Signed-off-by: Matt Wilson <msw@amazon.com>

diff -r 674b694814c8 -r 9a308e4fdc19 config/Tools.mk.in
--- a/config/Tools.mk.in	Wed Aug 29 11:07:52 2012 -0700
+++ b/config/Tools.mk.in	Wed Aug 29 11:50:44 2012 -0700
@@ -33,6 +33,8 @@
 DOT                 := @DOT@
 NEATO               := @NEATO@
 MARKDOWN            := @MARKDOWN@
+HTMLDUMP            := @HTMLDUMP@
+HTMLDUMPFLAGS       := @HTMLDUMPFLAGS@
 
 # Extra folder for libs/includes
 PREPEND_INCLUDES    := @PREPEND_INCLUDES@
diff -r 674b694814c8 -r 9a308e4fdc19 docs/Docs.mk
--- a/docs/Docs.mk	Wed Aug 29 11:07:52 2012 -0700
+++ b/docs/Docs.mk	Wed Aug 29 11:50:44 2012 -0700
@@ -10,3 +10,5 @@
 DOT		:= dot
 NEATO		:= neato
 MARKDOWN	:= markdown
+HTMLDUMP        := elinks
+HTMLDUMPFLAGS   := -dump
diff -r 674b694814c8 -r 9a308e4fdc19 docs/Makefile
--- a/docs/Makefile	Wed Aug 29 11:07:52 2012 -0700
+++ b/docs/Makefile	Wed Aug 29 11:50:44 2012 -0700
@@ -136,9 +136,17 @@
 	$(call move-if-changed,$@.tmp,$@)
 
 txt/%.txt: %.markdown
-	$(INSTALL_DIR) $(@D)
-	cp $< $@.tmp
-	$(call move-if-changed,$@.tmp,$@)
+	@$(INSTALL_DIR) $(@D)
+	set -e ; \
+	if which $(MARKDOWN) >/dev/null 2>&1 && \
+		which $(HTMLDUMP) >/dev/null 2>&1 ; then \
+		echo "Running markdown to generate $*.txt ... "; \
+		$(MARKDOWN) $< | $(HTMLDUMP) $(HTMLDUMPFLAGS) > $@.tmp ; \
+		$(call move-if-changed,$@.tmp,$@) ; \
+	else \
+		echo "markdown or html dump tool (like links) not installed; just copying $<."; \
+		cp $< $@; \
+	fi
 
 txt/man/%.1.txt: man/%.pod.1 Makefile
 	$(INSTALL_DIR) $(@D)
diff -r 674b694814c8 -r 9a308e4fdc19 tools/configure.ac
--- a/tools/configure.ac	Wed Aug 29 11:07:52 2012 -0700
+++ b/tools/configure.ac	Wed Aug 29 11:50:44 2012 -0700
@@ -92,6 +92,16 @@
 AX_DOCS_TOOL_PROG([DOT], [dot])
 AX_DOCS_TOOL_PROG([NEATO], [neato])
 AX_DOCS_TOOL_PROG([MARKDOWN], [markdown])
+
+AC_ARG_VAR([HTMLDUMP],
+           [Path to html-to-text generation tool (default: elinks)])
+AC_PATH_PROG([HTMLDUMP], [elinks])
+AS_IF([! test -x "$ac_cv_path_HTMLDUMP"], [
+    AC_MSG_WARN([$ac_cv_path_HTMLDUMP is not available so text documentation will be unformatted markdown])
+])
+AC_SUBST([HTMLDUMPFLAGS], ["-dump"])
+AC_ARG_VAR([HTMLDUMPFLAGS], [Flags passed to html to text translation tool])
+
 AS_IF([test "x$xapi" = "xy"], [
     AX_PATH_PROG_OR_FAIL([CURL], [curl-config])
     AX_PATH_PROG_OR_FAIL([XML], [xml2-config])

[PATCH 3 of 3] tools/docs: allow markdown_py to be used

[Matt Wilson]

An alternative Python markdown implementation is available on some
systems as markdown_py, so look for that path as well.

Signed-off-by: Matt Wilson <msw@amazon.com>

diff -r 9a308e4fdc19 -r f5a57d912d9f tools/configure.ac
--- a/tools/configure.ac	Wed Aug 29 11:50:44 2012 -0700
+++ b/tools/configure.ac	Wed Aug 29 11:58:30 2012 -0700
@@ -91,7 +91,7 @@
 AX_DOCS_TOOL_PROG([POD2TEXT], [pod2text])
 AX_DOCS_TOOL_PROG([DOT], [dot])
 AX_DOCS_TOOL_PROG([NEATO], [neato])
-AX_DOCS_TOOL_PROG([MARKDOWN], [markdown])
+AX_DOCS_TOOL_PROGS([MARKDOWN], [markdown markdown_py])
 
 AC_ARG_VAR([HTMLDUMP],
            [Path to html-to-text generation tool (default: elinks)])
diff -r 9a308e4fdc19 -r f5a57d912d9f tools/m4/docs_tool.m4
--- a/tools/m4/docs_tool.m4	Wed Aug 29 11:50:44 2012 -0700
+++ b/tools/m4/docs_tool.m4	Wed Aug 29 11:58:30 2012 -0700
@@ -6,3 +6,12 @@
         AC_MSG_WARN([$2 is not available so some documentation won't be built])
     ])
 ])
+
+AC_DEFUN([AX_DOCS_TOOL_PROGS], [
+dnl
+    AC_ARG_VAR([$1], [Path to $2 tool])
+    AC_PATH_PROGS([$1], [$2])
+    AS_IF([! test -x "$ac_cv_path_$1"], [
+        AC_MSG_WARN([$2 is not available so some documentation won't be built])
+    ])
+])

Re: [PATCH 1 of 3] tools: check for documentation generation tools at configure time

[Ian Jackson]

Matt Wilson writes ("[Xen-devel] [PATCH 1 of 3] tools: check for documentation generation tools at configure time"):
> It is sometimes hard to discover all the optional documentation
> generation tools that should be on a system to build all available Xen
> documentation. By checking for documentation generation tools at
> ./configure time and displaying a warning, Xen packagers will more
> easily learn about new optional build dependencies, like markdown,
> when they are introduced.

The way you've done this seems a bit odd to me:

> diff -r d7e4efa17fb0 -r 674b694814c8 config/Tools.mk.in
> --- a/config/Tools.mk.in	Tue Aug 28 15:35:08 2012 -0700
> +++ b/config/Tools.mk.in	Wed Aug 29 11:07:52 2012 -0700
> @@ -22,6 +22,17 @@
>  LD86                := @LD86@
>  BCC                 := @BCC@
>  IASL                := @IASL@
> +PS2PDF              := @PS2PDF@
> +DVIPS               := @DVIPS@

But this leaves settings of PS2PDF := ps2pdf in docs/Docs.mk AFAICT.
Surely this should be all plumbed through to the same places ?
Or is this part of the fallback mechanism ?  It's not clear.

You do say this:

> diff -r d7e4efa17fb0 -r 674b694814c8 docs/Makefile
> --- a/docs/Makefile	Tue Aug 28 15:35:08 2012 -0700
> +++ b/docs/Makefile	Wed Aug 29 11:07:52 2012 -0700
> @@ -3,6 +3,11 @@
>  XEN_ROOT=$(CURDIR)/..
>  include $(XEN_ROOT)/Config.mk
>  include $(XEN_ROOT)/docs/Docs.mk
> +# The default documentation tools specified in Docs.mk can be
> +# persistently overridden by the user via ./configure, but running
> +# ./configure is not required to build the docs tree. Thus Tools.mk is
> +# optionally included.
> +-include $(XEN_ROOT)/config/Tools.mk

But I think setting the same thing in two places like this does need
to be documented more clearly.  In particular if I'm right about the
purpose of the two settings, each set needs a comment pointing at the
other so that (a) they can be both changed at once when either is
changed (b) people don't get confused when the setting they're
changing has no effect.

Ian.

Re: [PATCH 3 of 3] tools/docs: allow markdown_py to be used

[Ian Jackson]

Matt Wilson writes ("[Xen-devel] [PATCH 3 of 3] tools/docs: allow markdown_py to be used"):
> An alternative Python markdown implementation is available on some
> systems as markdown_py, so look for that path as well.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Good for 4.2.

Subject of course to the other patches going in.

Ian.

Re: [PATCH 2 of 3] docs: use elinks to format markdown-generated html to text

[Ian Jackson]

Matt Wilson writes ("[Xen-devel] [PATCH 2 of 3] docs: use elinks to format markdown-generated html to text"):
> Markdown, while easy to read and write, isn't the most consumable
> format for users reading documentation on a terminal. This patch uses
> lynx to format markdown produced HTML into text files.
...
>  txt/%.txt: %.markdown
> -	$(INSTALL_DIR) $(@D)
> -	cp $< $@.tmp
> -	$(call move-if-changed,$@.tmp,$@)
> +	@$(INSTALL_DIR) $(@D)
> +	set -e ; \
> +	if which $(MARKDOWN) >/dev/null 2>&1 && \
> +		which $(HTMLDUMP) >/dev/null 2>&1 ; then \
> +		echo "Running markdown to generate $*.txt ... "; \

So now we have two efforts to try to find markdown, one in configure
and one here.

Keir, would it be OK if we simply declared that you must run configure
to "make docs" ?

Ian.

Re: [PATCH 2 of 3] docs: use elinks to format markdown-generated html to text

[Keir Fraser]

On 30/08/2012 16:41, "Ian Jackson" <Ian.Jackson@eu.citrix.com> wrote:

> Matt Wilson writes ("[Xen-devel] [PATCH 2 of 3] docs: use elinks to format
> markdown-generated html to text"):
>> Markdown, while easy to read and write, isn't the most consumable
>> format for users reading documentation on a terminal. This patch uses
>> lynx to format markdown produced HTML into text files.
> ...
>>  txt/%.txt: %.markdown
>> - $(INSTALL_DIR) $(@D)
>> - cp $< $@.tmp
>> - $(call move-if-changed,$@.tmp,$@)
>> + @$(INSTALL_DIR) $(@D)
>> + set -e ; \
>> + if which $(MARKDOWN) >/dev/null 2>&1 && \
>> +  which $(HTMLDUMP) >/dev/null 2>&1 ; then \
>> +  echo "Running markdown to generate $*.txt ... "; \
> 
> So now we have two efforts to try to find markdown, one in configure
> and one here.
> 
> Keir, would it be OK if we simply declared that you must run configure
> to "make docs" ?

Yes!

 -- Keir

> Ian.

Re: [PATCH 2 of 3] docs: use elinks to format markdown-generated html to text

[Ian Campbell]

On Thu, 2012-08-30 at 16:41 +0100, Ian Jackson wrote:
> Matt Wilson writes ("[Xen-devel] [PATCH 2 of 3] docs: use elinks to format markdown-generated html to text"):
> > Markdown, while easy to read and write, isn't the most consumable
> > format for users reading documentation on a terminal. This patch uses
> > lynx to format markdown produced HTML into text files.
> ...
> >  txt/%.txt: %.markdown
> > -	$(INSTALL_DIR) $(@D)
> > -	cp $< $@.tmp
> > -	$(call move-if-changed,$@.tmp,$@)
> > +	@$(INSTALL_DIR) $(@D)
> > +	set -e ; \
> > +	if which $(MARKDOWN) >/dev/null 2>&1 && \
> > +		which $(HTMLDUMP) >/dev/null 2>&1 ; then \
> > +		echo "Running markdown to generate $*.txt ... "; \
> 
> So now we have two efforts to try to find markdown, one in configure
> and one here.

If we are going to have this idea that docs works without or without
running configure then I think it is fine for the without case to just
copy unconditionally and not run markdown.

> Keir, would it be OK if we simply declared that you must run configure
> to "make docs" ?

Also an option.

Ian

Re: [PATCH 2 of 3] docs: use elinks to format markdown-generated html to text

[Matt Wilson]

On Thu, Aug 30, 2012 at 04:47:29PM +0100, Keir Fraser wrote:
> On 30/08/2012 16:41, "Ian Jackson" <Ian.Jackson@eu.citrix.com> wrote:
> 
> > Matt Wilson writes ("[Xen-devel] [PATCH 2 of 3] docs: use elinks to format
> > markdown-generated html to text"):
> >> Markdown, while easy to read and write, isn't the most consumable
> >> format for users reading documentation on a terminal. This patch uses
> >> lynx to format markdown produced HTML into text files.
> > ...
> >>  txt/%.txt: %.markdown
> >> - $(INSTALL_DIR) $(@D)
> >> - cp $< $@.tmp
> >> - $(call move-if-changed,$@.tmp,$@)
> >> + @$(INSTALL_DIR) $(@D)
> >> + set -e ; \
> >> + if which $(MARKDOWN) >/dev/null 2>&1 && \
> >> +  which $(HTMLDUMP) >/dev/null 2>&1 ; then \
> >> +  echo "Running markdown to generate $*.txt ... "; \
> > 
> > So now we have two efforts to try to find markdown, one in configure
> > and one here.
> > 
> > Keir, would it be OK if we simply declared that you must run configure
> > to "make docs" ?
> 
> Yes!

Great! The confusing bit of having these tools in two places was only
to retain the ability to run 'make -C docs' without running
./configure. I'll resubmit with this cleaned up.

Matt

Re: [PATCH 2 of 3] docs: use elinks to format markdown-generated html to text

[Matt Wilson]

On Thu, Aug 30, 2012 at 04:41:08PM +0100, Ian Jackson wrote:
> Matt Wilson writes ("[Xen-devel] [PATCH 2 of 3] docs: use elinks to format markdown-generated html to text"):
> > Markdown, while easy to read and write, isn't the most consumable
> > format for users reading documentation on a terminal. This patch uses
> > lynx to format markdown produced HTML into text files.
> ...
> >  txt/%.txt: %.markdown
> > -	$(INSTALL_DIR) $(@D)
> > -	cp $< $@.tmp
> > -	$(call move-if-changed,$@.tmp,$@)
> > +	@$(INSTALL_DIR) $(@D)
> > +	set -e ; \
> > +	if which $(MARKDOWN) >/dev/null 2>&1 && \
> > +		which $(HTMLDUMP) >/dev/null 2>&1 ; then \
> > +		echo "Running markdown to generate $*.txt ... "; \
> 
> So now we have two efforts to try to find markdown, one in configure
> and one here.

For what it's worth, if ./configure is run and a markdown binary is
found in $PATH, $(MARKDOWN) will be the full path detected at
./configure time like /usr/bin/markdown and the "which" bit will be a
no-op.

> Keir, would it be OK if we simply declared that you must run configure
> to "make docs" ?
> 
> Ian.