[RFC PATCH v3 2/3 (amend)] Documentation: complete config list from other manpages

[Jakub Narebski <jnareb@gmail.com>]

From: Thomas Rast <trast@student.ethz.ch>

Add an autogeneration script Documentation/make-config-list.perl
that complete list of config variables with missing variables
from other manpages.

This script generates minimal documentation for those missing config
variables at appropriate place in Documentation/config-vars.txt,
using the following form:

    foo.bar::
    foo.baz::
        See linkgit:git-foo[1].


It does that as follows:

* parse config-vars-src.txt (was config-vars.txt, which is now generated)
  to find out config variables it contains

* parse each manpage source (following includes) for config variable
  headers

* assemble a new config-vars.txt that completes the original list with
  "See linkgit:git-foo[1]" entries for all variables that were not in
  it config-vars-src.txt

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
This version has removed stray commented out remains of debugging the
make-config-list.perl script, and has it slightly reordered for better
readibility.

I have accidentally included old version of a patch.  I am very sorry.

Below there comments from previous email (without references to
attachements, which are not included in this email)
...

The differences from v2 by Thomas Rast:
* Removed stray changes to Documentation/Makefile and 
  Documentation/config-vars-src.txt

* The config-vars.txt target now depends on $(cmds_txt), which are 
  included by git.txt (without it, and without 'make doc' ran, the
  generation of "make -C Documentation config-vars.txt" failed because
  it couldn't find IIRC cmds-mainporcelain.txt).  See below.

* The config-vars.txt target now uses $(QUIET_GEN) and $(PERL_PATH),
  and makes use of make's automatic variables ($@, $<), like other
  targets that run *.perl scripts.  It uses $(MAN_TXT) in place of
  its definition, i.e. '$(MAN1_TXT) $(MAN5_TXT) $(MAN7_TXT)'

* The make-config-vars.perl invocation in config-vars.txt target
  now has extra '--ignore=merge-config.txt' because config-vars.src.txt
  contains 'include::merge-config.txt[]'.  See below.

* The make-config-vars.perl got rewritten according to proposals in
  parent email.  In general this means that instead of decomposing
  config-vars-src.txt, adding documentation of missing variables,
  and then recomposing it into config-vars.txt (with side-effects
  such as lowercasing variable names, and sorting variables), it
  simply finds places where to insert missing documentation, and
  generates and inserts it there.

  This means that read_varlist() got simplified, and main code got
  rewritten.


Issues to be solved (aka why this is an RFC):
* Dependencies for config-vars.txt target; probably just needs 
  modification to build-docdep.perl script.

* read_varlist does not follow includes, and that is why we had to
  manually ignore merge-config.txt

* Either sorting or inserting generated documentation could be
  improved; as can be seen in attached config-vars.txt the variables
  from manpage for git-http-backend got split.

 Documentation/Makefile                             |    5 +
 .../{config-vars.txt => config-vars-src.txt}       |    0
 Documentation/make-config-list.perl                |  168 ++++++++++++++++++++
 3 files changed, 173 insertions(+), 0 deletions(-)
 rename Documentation/{config-vars.txt => config-vars-src.txt} (100%)
 create mode 100755 Documentation/make-config-list.perl

diff --git a/Documentation/Makefile b/Documentation/Makefile
index e117bc4..8ce75d2 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -320,6 +320,11 @@ howto-index.txt: howto-index.sh $(wildcard howto/*.txt)
 	'$(SHELL_PATH_SQ)' ./howto-index.sh $(wildcard howto/*.txt) >$@+ && \
 	mv $@+ $@
 
+config-vars.txt: config-vars-src.txt $(MAN_TXT) $(cmds_txt)
+	$(QUIET_GEN)$(PERL_PATH) ./make-config-list.perl \
+		--mainlist=$< --ignore=$@ --ignore=merge-config.txt $(MAN_TXT) >$@+ && \
+	mv $@+ $@
+
 $(patsubst %,%.html,$(ARTICLES)) : %.html : %.txt
 	$(QUIET_ASCIIDOC)$(ASCIIDOC) $(ASCIIDOC_EXTRA) -b xhtml11 $*.txt
 
diff --git a/Documentation/config-vars.txt b/Documentation/config-vars-src.txt
similarity index 100%
rename from Documentation/config-vars.txt
rename to Documentation/config-vars-src.txt
diff --git a/Documentation/make-config-list.perl b/Documentation/make-config-list.perl
new file mode 100755
index 0000000..2894d96
--- /dev/null
+++ b/Documentation/make-config-list.perl
@@ -0,0 +1,168 @@
+#!/usr/bin/perl
+
+use strict;
+use warnings;
+use Getopt::Long;
+
+
+my %ignore;
+my $rc = GetOptions(
+	"mainlist=s" => \my $mainlistfile,
+	"ignore=s" => sub { $ignore{$_[1]} = 1 },
+);
+
+if (!$rc || !defined $mainlistfile) {
+	print "$0 --mainlist=<mainlist> [--ignore=<ignore>...] <asciidoc_manpage>...\n";
+	exit 1;
+}
+
+my %var_manpages;
+my %manpage_section;
+
+foreach my $name (@ARGV) {
+	read_man_txt($name);
+}
+
+my ($mainlist, $mainvars) = read_varlist($mainlistfile);
+
+my @missing_vars =
+	grep { !exists $mainlist->{lc($_)} } keys %var_manpages;
+my %missing_vars =
+	map { $_ => $var_manpages{$_} } @missing_vars;
+
+my %insert = find_insertion_points($mainlist, \%missing_vars);
+
+open my $fh, '<', $mainlistfile
+	or die "Couldn't open '$mainlistfile' for reading: $!";
+while (<$fh>) {
+	if (exists $insert{$.}) {
+		print vars_documentation($insert{$.}, \%missing_vars);
+		print "\n";
+	}
+	print;
+}
+# special case: insertion after last line in $mainlistfile
+print vars_documentation($insert{-1}, \%missing_vars)
+	if exists $insert{-1};
+close $fh
+	or die "Couldn't close '$mainlistfile': $!";
+
+exit 0;
+
+# ----------------------------------------------------------------------
+# ----------------------------------------------------------------------
+# ----------------------------------------------------------------------
+
+sub read_varlist {
+	my ($filename) = @_;
+
+	open my $fh, '<', $filename
+		or die "Couldn't open '$filename' for reading: $!";
+
+	my (%mainlist, @mainvars);
+	while (<$fh>) {
+		if (/^(\S+)::/) {
+			my $v = $1;
+			push @mainvars, $v;
+			$mainlist{lc($v)} = $.;
+		}
+	}
+
+	close $fh
+		or die "Couldn't close '$filename': $!";
+
+	return \%mainlist, \@mainvars;
+}
+
+sub read_man_txt {
+	my ($filename, $manpage) = @_;
+	if (!defined $manpage) {
+		$manpage = $filename;
+		$manpage =~ s/\.txt//;
+	}
+
+	open my $fh, '<', $filename
+		or die "Couldn't open '$filename' for reading: $!";
+	while (<$fh>) {
+		if ($. < 5 && /^$manpage\((\d+)\)/) {
+			$manpage_section{$manpage} = $1;
+		}
+		if (/^([a-z0-9]+\.[a-zA-Z<>0-9.]+)::/) {
+			push @{$var_manpages{$1}}, $manpage;
+		}
+		if (/^include::\s*(\S+)\s*\[\]/ &&
+		    !exists $ignore{$1}) {
+			read_man_txt($1, $manpage);
+		}
+	}
+	close $fh
+		or die "Couldn't close '$filename': $!";
+}
+
+sub find_insertion_points {
+	my ($mainlist, $missing_vars) = @_;
+	my %insert;
+
+	my %all_vars = (%$mainlist, %$missing_vars);
+	my $lineno = -1; # means after last line
+
+	# reverse order because we want to find a place before which to insert
+	# generated documentation; it is easy to find where description
+	# of variable begins, but in general harder to find where it ends.
+	my @sorted_vars = reverse sort { lc($a) cmp lc($b) } keys %all_vars;
+	foreach my $key (@sorted_vars) {
+		my $val = $all_vars{$key};
+		if (ref $val) {
+			# this came from %$missing_vars
+			push @{$insert{$lineno}}, $key;
+		} else {
+			# this came from %$mainlist
+			if ($lineno < 0) {
+				# $lineno < 0 means after end of file (special case)
+				$lineno = $val;
+			} else {
+				# this is in case of unsorted entries in $mainlistfile
+				$lineno = $val < $lineno ? $val : $lineno; # min($val, $lineno)
+			}
+		}
+	}
+	return %insert;
+}
+
+sub vars_documentation {
+	my ($keylist, $vars) = @_;
+	my @keys = sort @$keylist;
+	my %out;
+
+	# generate output for each key now, because it is easier to compare
+	# strings than arrays; comparing which is needed for compacting output
+	foreach my $k (@keys) {
+		$out{$k} = "\tSee: ".gen_links($vars->{$k}).".\n";
+	}
+
+	my $output = '';
+	while (my $k = pop @keys) {
+		$output .= $k."::\n";
+		unless (@keys && $out{$k} eq $out{$keys[0]}) {
+			$output .= $out{$k};
+		}
+	}
+	return $output;
+}
+
+sub gen_links {
+	my $manpages = shift;
+	return join(", ", map { linkgit($_) } @$manpages);
+}
+
+sub linkgit {
+	my $manpage = shift;
+
+	if (!exists $manpage_section{$manpage}) {
+		warn "section for $manpage unknown, assuming '1'\n";
+		$manpage_section{$manpage} = 1;
+	}
+	return "linkgit:${manpage}[$manpage_section{$manpage}]";
+}
+
+__END__
-- 
1.7.3