[PATCH] generate-cmdlist: re-implement as shell script

[PATCH] generate-cmdlist: re-implement as shell script

[Eric Sunshine]

527ec39 (generate-cmdlist: parse common group commands, 2015-05-21)
replaced generate-cmdlist.sh with a more functional Perl version,
generate-cmdlist.perl. The Perl version gleans named tags from a new
"common groups" section in command-list.txt and recognizes those tags in
"command list" section entries in place of the old 'common' tag. This
allows git-help to, not only recognize, but also group common commands.

Although the tests require Perl, 527ec39 creates an unconditional
dependence upon Perl in the build system itself, which can not be
overridden with NO_PERL. Such a dependency may be undesirable; for
instance, the 'git-lite' package in the FreeBSD ports tree is intended
as a minimal Git installation (which may, for example, be useful on
servers needing only local clone and update capability), which,
historically, has not depended upon Perl[1].

Therefore, revive generate-cmdlist.sh and extend it to recognize "common
groups" and its named tags. Retire generate-cmdlist.perl.

[1]: http://thread.gmane.org/gmane.comp.version-control.git/275905/focus=276132

Signed-off-by: Eric Sunshine <sunshine@sunshineco.com>
---

In addition to reviving 527ec39^:generate-cmdlist.sh and extending it, I
also re-indented it with tabs instead of spaces, so it's helpful to
ignore whitespace changes when comparing the old and new versions of the
shell script.

Makefile              |  4 ++--
 generate-cmdlist.perl | 50 --------------------------------------------------
 generate-cmdlist.sh   | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 52 insertions(+), 52 deletions(-)
 delete mode 100755 generate-cmdlist.perl
 create mode 100755 generate-cmdlist.sh

diff --git a/Makefile b/Makefile
index e39ca6c..ddfe7a1 100644
--- a/Makefile
+++ b/Makefile
@@ -1697,10 +1697,10 @@ $(BUILT_INS): git$X
 	ln -s $< $@ 2>/dev/null || \
 	cp $< $@
 
-common-cmds.h: generate-cmdlist.perl command-list.txt
+common-cmds.h: generate-cmdlist.sh command-list.txt
 
 common-cmds.h: $(wildcard Documentation/git-*.txt)
-	$(QUIET_GEN)$(PERL_PATH) generate-cmdlist.perl command-list.txt > $@+ && mv $@+ $@
+	$(QUIET_GEN)./generate-cmdlist.sh <command-list.txt >$@+ && mv $@+ $@
 
 SCRIPT_DEFINES = $(SHELL_PATH_SQ):$(DIFF_SQ):$(GIT_VERSION):\
 	$(localedir_SQ):$(NO_CURL):$(USE_GETTEXT_SCHEME):$(SANE_TOOL_PATH_SQ):\
diff --git a/generate-cmdlist.perl b/generate-cmdlist.perl
deleted file mode 100755
index 31516e3..0000000
--- a/generate-cmdlist.perl
+++ /dev/null
@@ -1,50 +0,0 @@
-#!/usr/bin/perl
-use strict;
-use warnings;
-
-print <<"EOT";
-/* Automatically generated by $0 */
-
-struct cmdname_help {
-	char name[16];
-	char help[80];
-	unsigned char group;
-};
-
-static char *common_cmd_groups[] = {
-EOT
-
-my $n = 0;
-my %grp;
-while (<>) {
-	last if /^### command list/;
-	next if (1../^### common groups/) || /^#/ || /^\s*$/;
-	chop;
-	my ($k, $v) = split ' ', $_, 2;
-	$grp{$k} = $n++;
-	print "\tN_(\"$v\"),\n";
-}
-
-print "};\n\nstatic struct cmdname_help common_cmds[] = {\n";
-
-while (<>) {
-	next if /^#/ || /^\s*$/;
-	my @tags = split;
-	my $cmd = shift @tags;
-	for my $t (@tags) {
-		if (exists $grp{$t}) {
-			my $s;
-			open my $f, '<', "Documentation/$cmd.txt" or die;
-			while (<$f>) {
-				($s) = /^$cmd - (.+)$/;
-				last if $s;
-			}
-			close $f;
-			$cmd =~ s/^git-//;
-			print "\t{\"$cmd\", N_(\"$s\"), $grp{$t}},\n";
-			last;
-		}
-	}
-}
-
-print "};\n";
diff --git a/generate-cmdlist.sh b/generate-cmdlist.sh
new file mode 100755
index 0000000..1ac329d
--- /dev/null
+++ b/generate-cmdlist.sh
@@ -0,0 +1,50 @@
+#!/bin/sh
+
+echo "/* Automatically generated by $0 */
+struct cmdname_help {
+	char name[16];
+	char help[80];
+	unsigned char group;
+};
+
+static const char *common_cmd_groups[] = {"
+
+tmp=cmdgrps$$.tmp
+trap "rm -fr $tmp" 0 1 2 3 15
+
+sed -n '
+	1,/^### common groups/b
+	/^### command list/q
+	/^#/d; /^[ 	]*$/b
+	h;s/^[^ 	][^ 	]*[ 	][ 	]*\(.*\)/	N_("\1"),/p
+	g;s/^\([^ 	][^ 	]*\)[ 	].*/\1/w '$tmp'
+	'
+printf '};\n\n'
+
+n=0
+matchgrp=
+substnum=
+while read grp
+do
+	matchgrp="$matchgrp${matchgrp:+
+}^git-..*[ 	]$grp"
+	substnum="$substnum${substnum:+;}s/[ 	]$grp/$n/"
+	n=$(($n+1))
+done <$tmp
+
+printf 'static struct cmdname_help common_cmds[] = {\n'
+grep "$matchgrp" |
+sed 's/^git-//' |
+sort |
+while read cmd tags
+do
+	tag=$(echo $tags | sed "$substnum; s/[^0-9]//g")
+	sed -n '
+		/^NAME/,/git-'"$cmd"'/H
+		${
+			x
+			s/.*git-'"$cmd"' - \(.*\)/	{"'"$cmd"'", N_("\1"), '$tag'},/
+			p
+		}' "Documentation/git-$cmd.txt"
+done
+echo "};"
-- 
2.5.0.457.gab17608

Re: [PATCH] generate-cmdlist: re-implement as shell script

[Junio C Hamano]

Eric Sunshine <sunshine@sunshineco.com> writes:

> In addition to reviving 527ec39^:generate-cmdlist.sh and extending it, I
> also re-indented it with tabs instead of spaces, so it's helpful to
> ignore whitespace changes when comparing the old and new versions of the
> shell script.

Hmph.  Perhaps we can view it as part of reverting 527ec39 and then
redoing it in shell.  The way the shell script accumulates matchgrp
variable (i.e. the literal LF in ${var:+string}) makes me feel some
possible portability scare, which might be solved in a more stupid
(i.e. not giving various reimplementations of Bourne shells a chance
to screw it up) way by using another temporary file, but other than
that the resurrected script looks OK to me.

Thanks.

> diff --git a/generate-cmdlist.sh b/generate-cmdlist.sh
> new file mode 100755
> index 0000000..1ac329d
> --- /dev/null
> +++ b/generate-cmdlist.sh
> @@ -0,0 +1,50 @@
> +#!/bin/sh
> +
> +echo "/* Automatically generated by $0 */
> +struct cmdname_help {
> +	char name[16];
> +	char help[80];
> +	unsigned char group;
> +};
> +
> +static const char *common_cmd_groups[] = {"
> +
> +tmp=cmdgrps$$.tmp
> +trap "rm -fr $tmp" 0 1 2 3 15
> +
> +sed -n '
> +	1,/^### common groups/b
> +	/^### command list/q
> +	/^#/d; /^[ 	]*$/b
> +	h;s/^[^ 	][^ 	]*[ 	][ 	]*\(.*\)/	N_("\1"),/p
> +	g;s/^\([^ 	][^ 	]*\)[ 	].*/\1/w '$tmp'
> +	'
> +printf '};\n\n'
> +
> +n=0
> +matchgrp=
> +substnum=
> +while read grp
> +do
> +	matchgrp="$matchgrp${matchgrp:+
> +}^git-..*[ 	]$grp"
> +	substnum="$substnum${substnum:+;}s/[ 	]$grp/$n/"
> +	n=$(($n+1))
> +done <$tmp
> +
> +printf 'static struct cmdname_help common_cmds[] = {\n'
> +grep "$matchgrp" |
> +sed 's/^git-//' |
> +sort |
> +while read cmd tags
> +do
> +	tag=$(echo $tags | sed "$substnum; s/[^0-9]//g")
> +	sed -n '
> +		/^NAME/,/git-'"$cmd"'/H
> +		${
> +			x
> +			s/.*git-'"$cmd"' - \(.*\)/	{"'"$cmd"'", N_("\1"), '$tag'},/
> +			p
> +		}' "Documentation/git-$cmd.txt"
> +done
> +echo "};"

Re: [PATCH] generate-cmdlist: re-implement as shell script

[Eric Sunshine]

On Wed, Aug 19, 2015 at 6:38 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Eric Sunshine <sunshine@sunshineco.com> writes:
>> In addition to reviving 527ec39^:generate-cmdlist.sh and extending it, I
>> also re-indented it with tabs instead of spaces, so it's helpful to
>> ignore whitespace changes when comparing the old and new versions of the
>> shell script.
>
> Hmph.  Perhaps we can view it as part of reverting 527ec39 and then
> redoing it in shell.

I'm having trouble understanding. Are you asking that this be
presented as a series of patches which first revert 527ec39, then do
the whitespace cleanup, and then augment the script for the extended
functionality? If so, it's a bit problematic because the original
script still expects the 'common' tag to be present, but that tag was
removed by the subsequent patch 2f5b495 (command-list.txt: drop the
"common" tag, 2015-05-21). So, reverting 527ec39 would also require
reverting 2f5b495.

> The way the shell script accumulates matchgrp
> variable (i.e. the literal LF in ${var:+string}) makes me feel some
> possible portability scare, which might be solved in a more stupid
> (i.e. not giving various reimplementations of Bourne shells a chance
> to screw it up) way by using another temporary file[...]

In addition to the literal newline and the temporary file, other
options I considered included assigning the newline to a variable and
then interpolating that variable (${var:+$LF}), or expanding the list
of patterns into a set of '-e' arguments for grep. But, I think I'll
just go with the temporary file.

Re: [PATCH] generate-cmdlist: re-implement as shell script

[Junio C Hamano]

Eric Sunshine <sunshine@sunshineco.com> writes:

> diff --git a/generate-cmdlist.sh b/generate-cmdlist.sh
> new file mode 100755
> index 0000000..1ac329d
> --- /dev/null
> +++ b/generate-cmdlist.sh
> @@ -0,0 +1,50 @@
> +#!/bin/sh
> +
> +echo "/* Automatically generated by $0 */
> +struct cmdname_help {
> +	char name[16];
> +	char help[80];
> +	unsigned char group;
> +};
> +
> +static const char *common_cmd_groups[] = {"
> +
> +tmp=cmdgrps$$.tmp
> +trap "rm -fr $tmp" 0 1 2 3 15
> +
> +sed -n '
> +	1,/^### common groups/b
> +	/^### command list/q
> +	/^#/d; /^[ 	]*$/b
> +	h;s/^[^ 	][^ 	]*[ 	][ 	]*\(.*\)/	N_("\1"),/p
> +	g;s/^\([^ 	][^ 	]*\)[ 	].*/\1/w '$tmp'
> +	'
> +printf '};\n\n'

Unfortunately, this does not seem to work for me.  Even though sed
stops reading after seeing the "### command list" line, I suspect
that its stdin buffer has been filled with other lines that follow
it from the input to the buffer size, consuming what you meant to
feed the later 'grep $matchgrp"' with.

This is a one-time thing, so I do not mind to update the Makefile
so that it does not feed command-list.txt from the standard input
but gives the path as "$1" to this script.

> +n=0
> +matchgrp=
> +substnum=
> +while read grp
> +do
> +	matchgrp="$matchgrp${matchgrp:+
> +}^git-..*[ 	]$grp"
> +	substnum="$substnum${substnum:+;}s/[ 	]$grp/$n/"
> +	n=$(($n+1))
> +done <$tmp
> +
> +printf 'static struct cmdname_help common_cmds[] = {\n'
> +grep "$matchgrp" |
> +sed 's/^git-//' |
> +sort |
> +while read cmd tags
> +do
> +	tag=$(echo $tags | sed "$substnum; s/[^0-9]//g")
> +	sed -n '
> +		/^NAME/,/git-'"$cmd"'/H
> +		${
> +			x
> +			s/.*git-'"$cmd"' - \(.*\)/	{"'"$cmd"'", N_("\1"), '$tag'},/
> +			p
> +		}' "Documentation/git-$cmd.txt"
> +done
> +echo "};"

Re: [PATCH] generate-cmdlist: re-implement as shell script

[Eric Sunshine]

On Thu, Aug 20, 2015 at 1:24 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Eric Sunshine <sunshine@sunshineco.com> writes:
>> +sed -n '
>> +     1,/^### common groups/b
>> +     /^### command list/q
>> +     /^#/d; /^[      ]*$/b
>> +     h;s/^[^         ][^     ]*[     ][      ]*\(.*\)/       N_("\1"),/p
>> +     g;s/^\([^       ][^     ]*\)[   ].*/\1/w '$tmp'
>> +     '
>> +printf '};\n\n'
>
> Unfortunately, this does not seem to work for me.  Even though sed
> stops reading after seeing the "### command list" line, I suspect
> that its stdin buffer has been filled with other lines that follow
> it from the input to the buffer size, consuming what you meant to
> feed the later 'grep $matchgrp"' with.
>
> This is a one-time thing, so I do not mind to update the Makefile
> so that it does not feed command-list.txt from the standard input
> but gives the path as "$1" to this script.

The original generate-cmdlist.sh doesn't take any arguments and just
hardcodes "command-list.txt". Feeding it instead on stdin seemed a
nice way to avoid reading the file twice, but alas is too fragile. I
don't mind passing it as an argument either, or just hardcoding it
again (though my preference leans toward passing it as an argument).