More 'shortlog' statistics models?

More 'shortlog' statistics models?

[Linus Torvalds]

So I was just thinking about one of the things I've wondered about for a 
while: when do I (and others) make most of my commits?

I mean literally queries about time-of-day or day-of-week issues. I know 
we have the information, and I can write silly scripts to get it, but I'm 
wondering if somebody would actually be willing to make our current 
'shortlog' statistics gatherer do it automatically.

Right now shortlog groups by name, and I'd like to group by other things.

Here's an example of what I just played around with:

	git log --since=6.months --pretty=format:%cD --author=torvalds |
		cut -d, -f1 |
		sort |
		uniq -c |
		sort -n

gives you the weekdays on which I've done most of my commits in the last 
six months. And I think it's kind of interesting. I really do calm down a 
bit on weekends, and do only about half as many commits on Saturday and 
Sunday as I do during the week.

The same is true across the board, btw. Except while _I_ tend to do most 
of my commits on Mondays, while on average, Linux kernel people seem to do 
most of them on Fridays.

But what about time zones? Change the 'cut' to use "-d' ' -f6" instead, 
and it's clear that -0700 is the top time zone (US West coast), but +0100 
and +0200 (much of Europe) are pretty active.

How about time? "cut -d' ' -f5 | cut -d: -f1": The least productive time 
hour is 05:xxAM (not a big surprise, perhaps), while the most productive 
time is apparently early afternoon. But there I'm odd: most of my commits 
by far are in the morning (ie 8AM to noon).

Anyway, I can do all this with sorting, but it's fun enough that I wonder 
if we should make it easier?

		Linus

Re: More 'shortlog' statistics models?

[Junio C Hamano]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> So I was just thinking about one of the things I've wondered about for a 
> while: when do I (and others) make most of my commits?

I can see that I am more active on Wednesdays and Saturdays for obvious
reasons.  Thanks for an interesting diversion.

> Anyway, I can do all this with sorting, but it's fun enough that I wonder 
> if we should make it easier?

Wasn't one of the GSoC project about git statistics?  Or was it a topic of
last year?

[PATCH 1/2] shortlog: refactor insert_one_record()

[Johannes Schindelin]

This just splits insert_one_record() into one part which assumes that
the key is an email ident of the form "A U Thor <author@example.com>"
(that needs to respect .mailmap) and a second part which does not.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 builtin-shortlog.c |   81 ++++++++++++++++++++++++++++-----------------------
 1 files changed, 44 insertions(+), 37 deletions(-)

diff --git a/builtin-shortlog.c b/builtin-shortlog.c
index 6a3812e..a684422 100644
--- a/builtin-shortlog.c
+++ b/builtin-shortlog.c
@@ -32,19 +32,57 @@ static int compare_by_number(const void *a1, const void *a2)
 const char *format_subject(struct strbuf *sb, const char *msg,
 			   const char *line_separator);
 
-static void insert_one_record(struct shortlog *log,
-			      const char *author,
-			      const char *oneline)
+static void insert_one_record1(struct shortlog *log,
+		const char *name, const char *oneline)
 {
 	const char *dot3 = log->common_repo_prefix;
 	char *buffer, *p;
 	struct string_list_item *item;
+	const char *eol;
+	struct strbuf subject = STRBUF_INIT;
+
+	item = string_list_insert(name, &log->list);
+	if (item->util == NULL)
+		item->util = xcalloc(1, sizeof(struct string_list));
+
+	/* Skip any leading whitespace, including any blank lines. */
+	while (*oneline && isspace(*oneline))
+		oneline++;
+	eol = strchr(oneline, '\n');
+	if (!eol)
+		eol = oneline + strlen(oneline);
+	if (!prefixcmp(oneline, "[PATCH")) {
+		char *eob = strchr(oneline, ']');
+		if (eob && (!eol || eob < eol))
+			oneline = eob + 1;
+	}
+	while (*oneline && isspace(*oneline) && *oneline != '\n')
+		oneline++;
+	format_subject(&subject, oneline, " ");
+	buffer = strbuf_detach(&subject, NULL);
+
+	if (dot3) {
+		int dot3len = strlen(dot3);
+		if (dot3len > 5) {
+			while ((p = strstr(buffer, dot3)) != NULL) {
+				int taillen = strlen(p) - dot3len;
+				memcpy(p, "/.../", 5);
+				memmove(p + 5, p + dot3len, taillen + 1);
+			}
+		}
+	}
+
+	string_list_append(buffer, item->util);
+}
+
+static void insert_one_record(struct shortlog *log,
+			      const char *author,
+			      const char *oneline)
+{
 	char namebuf[1024];
 	char emailbuf[1024];
 	size_t len;
-	const char *eol;
 	const char *boemail, *eoemail;
-	struct strbuf subject = STRBUF_INIT;
 
 	boemail = strchr(author, '<');
 	if (!boemail)
@@ -84,38 +122,7 @@ static void insert_one_record(struct shortlog *log,
 		snprintf(namebuf + len, room, " <%.*s>", maillen, emailbuf);
 	}
 
-	item = string_list_insert(namebuf, &log->list);
-	if (item->util == NULL)
-		item->util = xcalloc(1, sizeof(struct string_list));
-
-	/* Skip any leading whitespace, including any blank lines. */
-	while (*oneline && isspace(*oneline))
-		oneline++;
-	eol = strchr(oneline, '\n');
-	if (!eol)
-		eol = oneline + strlen(oneline);
-	if (!prefixcmp(oneline, "[PATCH")) {
-		char *eob = strchr(oneline, ']');
-		if (eob && (!eol || eob < eol))
-			oneline = eob + 1;
-	}
-	while (*oneline && isspace(*oneline) && *oneline != '\n')
-		oneline++;
-	format_subject(&subject, oneline, " ");
-	buffer = strbuf_detach(&subject, NULL);
-
-	if (dot3) {
-		int dot3len = strlen(dot3);
-		if (dot3len > 5) {
-			while ((p = strstr(buffer, dot3)) != NULL) {
-				int taillen = strlen(p) - dot3len;
-				memcpy(p, "/.../", 5);
-				memmove(p + 5, p + dot3len, taillen + 1);
-			}
-		}
-	}
-
-	string_list_append(buffer, item->util);
+	insert_one_record1(log, namebuf, oneline);
 }
 
 static void read_from_stdin(struct shortlog *log)
-- 
1.6.3.284.g6fecc

[PATCH 2/2] shortlog: add '--sort-key' and '--sort-key-regexp' options

[Johannes Schindelin]

Instead of always sorting by author, allow to sort by either
a whitespace-delimited field or by a regular expression (first group)
on the oneline.

For example, this will give you an overview of the weekday/commit
distribution:

	git shortlog -k 1 --pretty=%cD -s -n

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---

	Pretty obviously, in git.git this shows that Saturday and
	Wednesday, your designated Git days, are on top of the list, but
	what is interesting to me is that the difference between Wednesday 
	and Sunday is not all that much.

	Sorry, no time for tests, maybe somebody else can help out?

 Documentation/git-shortlog.txt |   13 +++++++-
 builtin-shortlog.c             |   64 +++++++++++++++++++++++++++++++++++++--
 shortlog.h                     |    1 +
 3 files changed, 73 insertions(+), 5 deletions(-)

diff --git a/Documentation/git-shortlog.txt b/Documentation/git-shortlog.txt
index 42463a9..3155361 100644
--- a/Documentation/git-shortlog.txt
+++ b/Documentation/git-shortlog.txt
@@ -9,7 +9,7 @@ SYNOPSIS
 --------
 [verse]
 git log --pretty=short | 'git shortlog' [-h] [-n] [-s] [-e] [-w]
-git shortlog [-n|--numbered] [-s|--summary] [-e|--email] [-w[<width>[,<indent1>[,<indent2>]]]] [<committish>...]
+git shortlog [-n|--numbered] [-s|--summary] [-e|--email] [-w[<width>[,<indent1>[,<indent2>]]]] [(-K|--sort-key-regexp) <regexp>] [(-k|--sort-key) <n>] [<committish>...]
 
 DESCRIPTION
 -----------
@@ -45,6 +45,17 @@ OPTIONS
 	and subsequent lines are indented by `indent2` spaces. `width`,
 	`indent1`, and `indent2` default to 76, 6 and 9 respectively.
 
+-K <regexp>::
+--sort-key-regexp <regexp>::
+	Instead of sorting by author, sort by a regular expression on the
+	commit subject (or whatever you specified using --pretty)
+
+-k <number>::
+--sort-key <number>::
+	Instead of sorting by author, sort by a given whitespace-delimited
+	field of the commit subject (or whatever you specified using
+	--pretty).  The first field is 1.
+
 
 MAPPING AUTHORS
 ---------------
diff --git a/builtin-shortlog.c b/builtin-shortlog.c
index a684422..2cab5e4 100644
--- a/builtin-shortlog.c
+++ b/builtin-shortlog.c
@@ -145,9 +145,37 @@ static void read_from_stdin(struct shortlog *log)
 
 void shortlog_add_commit(struct shortlog *log, struct commit *commit)
 {
+	struct strbuf buf = STRBUF_INIT;
+	char *key = NULL;
 	const char *author = NULL, *buffer;
 
 	buffer = commit->buffer;
+	if (log->user_format)
+		pretty_print_commit(CMIT_FMT_USERFORMAT, commit, &buf,
+			DEFAULT_ABBREV, "", "", DATE_NORMAL, 0);
+
+	if (log->sort_key) {
+		const char *p = buf.buf;
+		regmatch_t match[2];
+
+		if (!log->user_format) {
+			p = strstr(buffer, "\n\n");
+			if (!p)
+				return;
+			p += 2;
+		}
+
+		if (!regexec(log->sort_key, p, 2, match, 0) &&
+				match[1].rm_so >= 0)
+			author = key = xstrndup(p + match[1].rm_so,
+				match[1].rm_eo - match[1].rm_so);
+		else
+			author = "<null>";
+		insert_one_record1(log, key, p);
+		strbuf_release(&buf);
+		return;
+	}
+
 	while (*buffer && *buffer != '\n') {
 		const char *eol = strchr(buffer, '\n');
 
@@ -164,10 +192,6 @@ void shortlog_add_commit(struct shortlog *log, struct commit *commit)
 		die("Missing author: %s",
 		    sha1_to_hex(commit->object.sha1));
 	if (log->user_format) {
-		struct strbuf buf = STRBUF_INIT;
-
-		pretty_print_commit(CMIT_FMT_USERFORMAT, commit, &buf,
-			DEFAULT_ABBREV, "", "", DATE_NORMAL, 0);
 		insert_one_record(log, author, buf.buf);
 		strbuf_release(&buf);
 		return;
@@ -175,6 +199,7 @@ void shortlog_add_commit(struct shortlog *log, struct commit *commit)
 	if (*buffer)
 		buffer++;
 	insert_one_record(log, author, !*buffer ? "<none>" : buffer);
+	free(key);
 }
 
 static void get_from_rev(struct rev_info *rev, struct shortlog *log)
@@ -250,6 +275,8 @@ int cmd_shortlog(int argc, const char **argv, const char *prefix)
 {
 	static struct shortlog log;
 	static struct rev_info rev;
+	static const char *sort_key_regexp = NULL;
+	static int sort_key_field = 0;
 	int nongit;
 
 	static const struct option options[] = {
@@ -261,6 +288,11 @@ int cmd_shortlog(int argc, const char **argv, const char *prefix)
 			    "Show the email address of each author"),
 		{ OPTION_CALLBACK, 'w', NULL, &log, "w[,i1[,i2]]",
 			"Linewrap output", PARSE_OPT_OPTARG, &parse_wrap_args },
+		OPT_STRING('K', "sort-key-regexp", &sort_key_regexp, "<regexp>",
+			"Sort shortlog by the given regular expression"),
+		OPT_INTEGER('k', "sort-key", &sort_key_field,
+			"Sort shortlog by the given field "
+			"(whitespace-delimited)"),
 		OPT_END(),
 	};
 
@@ -285,6 +317,27 @@ int cmd_shortlog(int argc, const char **argv, const char *prefix)
 parse_done:
 	argc = parse_options_end(&ctx);
 
+	if (sort_key_regexp && sort_key_field > 0)
+		die ("--sort-key-regexp and --sort-key-field are incompatible");
+
+	if (sort_key_regexp) {
+		log.sort_key = xmalloc(sizeof(*log.sort_key));
+		if (regcomp(log.sort_key, sort_key_regexp, 0))
+			die ("Invalid regular expression: '%s'",
+					sort_key_regexp);
+	}
+	if (sort_key_field > 0) {
+		struct strbuf regexp = STRBUF_INIT;
+		strbuf_addstr(&regexp, "^[ \t\n]*");
+		while (--sort_key_field)
+			strbuf_addstr(&regexp, "[^ \t\n]*[ \t\n]*");
+		strbuf_addstr(&regexp, "\\([^ \t\n]*\\)[ \t\n]*.*$");
+		log.sort_key = xmalloc(sizeof(*log.sort_key));
+		if (regcomp(log.sort_key, regexp.buf, 0))
+			die ("Invalid regular expression: '%s'", regexp.buf);
+		strbuf_release(&regexp);
+	}
+
 	if (setup_revisions(argc, argv, &rev, NULL) != 1) {
 		error("unrecognized argument: %s", argv[1]);
 		usage_with_options(shortlog_usage, options);
@@ -296,6 +349,9 @@ parse_done:
 	if (!nongit && !rev.pending.nr && isatty(0))
 		add_head_to_pending(&rev);
 	if (rev.pending.nr == 0) {
+		if (log.sort_key)
+			die ("Specifying a sort key is incompatible with "
+				"reading from stdin.");
 		read_from_stdin(&log);
 	}
 	else
diff --git a/shortlog.h b/shortlog.h
index bc02cc2..9dedaf6 100644
--- a/shortlog.h
+++ b/shortlog.h
@@ -16,6 +16,7 @@ struct shortlog {
 	char *common_repo_prefix;
 	int email;
 	struct string_list mailmap;
+	regex_t *sort_key;
 };
 
 void shortlog_init(struct shortlog *log);
-- 
1.6.3.284.g6fecc

Re: [PATCH 2/2] shortlog: add '--sort-key' and '--sort-key-regexp' options

[Linus Torvalds]

On Wed, 15 Jul 2009, Johannes Schindelin wrote:
> 
> Instead of always sorting by author, allow to sort by either
> a whitespace-delimited field or by a regular expression (first group)
> on the oneline.

Ok, I got a SIGSEGV on the regexp, because I didn't react to the "first 
group" part. So I did

	git shortlog --pretty=%cD --sort-key-regex ' [0-9]*:' -ns

and it immediately segfaulted for me.

I realize that I _should_ have done

	git shortlog --pretty=%cD --sort-key-regex ' \([0-9]*\):' -ns

and then I get the expected hourly statistics, but it shouldn't have 
SIGSEGV'd either.

But cool small patch-series. This was exactly what I was looking for.

		Linus

Re: [PATCH 2/2] shortlog: add '--sort-key' and '--sort-key-regexp' options

[Michał Kiedrowicz]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

> 
> Instead of always sorting by author, allow to sort by either
> a whitespace-delimited field or by a regular expression (first group)
> on the oneline.
> 
> For example, this will give you an overview of the weekday/commit
> distribution:
> 
> 	git shortlog -k 1 --pretty=%cD -s -n
> 
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
> 
> 	Pretty obviously, in git.git this shows that Saturday and
> 	Wednesday, your designated Git days, are on top of the list,
> but what is interesting to me is that the difference between
> Wednesday and Sunday is not all that much.
> 
> 	Sorry, no time for tests, maybe somebody else can help out?
> 
>  Documentation/git-shortlog.txt |   13 +++++++-
>  builtin-shortlog.c             |   64
> +++++++++++++++++++++++++++++++++++++--
> shortlog.h                     |    1 + 3 files changed, 73
> insertions(+), 5 deletions(-)
> 
> diff --git a/Documentation/git-shortlog.txt
> b/Documentation/git-shortlog.txt index 42463a9..3155361 100644
> --- a/Documentation/git-shortlog.txt
> +++ b/Documentation/git-shortlog.txt
> @@ -9,7 +9,7 @@ SYNOPSIS
>  --------
>  [verse]
>  git log --pretty=short | 'git shortlog' [-h] [-n] [-s] [-e] [-w]
> -git shortlog [-n|--numbered] [-s|--summary] [-e|--email]
> [-w[<width>[,<indent1>[,<indent2>]]]] [<committish>...] +git shortlog
> [-n|--numbered] [-s|--summary] [-e|--email]
> [-w[<width>[,<indent1>[,<indent2>]]]] [(-K|--sort-key-regexp)
> <regexp>] [(-k|--sort-key) <n>] [<committish>...] DESCRIPTION
> ----------- @@ -45,6 +45,17 @@ OPTIONS
>  	and subsequent lines are indented by `indent2` spaces.
> `width`, `indent1`, and `indent2` default to 76, 6 and 9 respectively.
>  
> +-K <regexp>::
> +--sort-key-regexp <regexp>::
> +	Instead of sorting by author, sort by a regular expression
> on the
> +	commit subject (or whatever you specified using --pretty)
> +
> +-k <number>::
> +--sort-key <number>::
> +	Instead of sorting by author, sort by a given
> whitespace-delimited
> +	field of the commit subject (or whatever you specified using
> +	--pretty).  The first field is 1.
> +
>  
>  MAPPING AUTHORS
>  ---------------
> diff --git a/builtin-shortlog.c b/builtin-shortlog.c
> index a684422..2cab5e4 100644
> --- a/builtin-shortlog.c
> +++ b/builtin-shortlog.c
> @@ -145,9 +145,37 @@ static void read_from_stdin(struct shortlog *log)
>  
>  void shortlog_add_commit(struct shortlog *log, struct commit *commit)
>  {
> +	struct strbuf buf = STRBUF_INIT;
> +	char *key = NULL;
>  	const char *author = NULL, *buffer;
>  
>  	buffer = commit->buffer;
> +	if (log->user_format)
> +		pretty_print_commit(CMIT_FMT_USERFORMAT, commit,
> &buf,
> +			DEFAULT_ABBREV, "", "", DATE_NORMAL, 0);
> +
> +	if (log->sort_key) {
> +		const char *p = buf.buf;
> +		regmatch_t match[2];
> +
> +		if (!log->user_format) {
> +			p = strstr(buffer, "\n\n");
> +			if (!p)
> +				return;
> +			p += 2;
> +		}
> +
> +		if (!regexec(log->sort_key, p, 2, match, 0) &&
> +				match[1].rm_so >= 0)
> +			author = key = xstrndup(p + match[1].rm_so,
> +				match[1].rm_eo - match[1].rm_so);
> +		else
> +			author = "<null>";
> +		insert_one_record1(log, key, p);
> +		strbuf_release(&buf);

Missing free()?

> +		return;
> +	}
> +
>  	while (*buffer && *buffer != '\n') {
>  		const char *eol = strchr(buffer, '\n');
>  
> @@ -164,10 +192,6 @@ void shortlog_add_commit(struct shortlog *log,
> struct commit *commit) die("Missing author: %s",
>  		    sha1_to_hex(commit->object.sha1));
>  	if (log->user_format) {
> -		struct strbuf buf = STRBUF_INIT;
> -
> -		pretty_print_commit(CMIT_FMT_USERFORMAT, commit,
> &buf,
> -			DEFAULT_ABBREV, "", "", DATE_NORMAL, 0);
>  		insert_one_record(log, author, buf.buf);
>  		strbuf_release(&buf);
>  		return;
> @@ -175,6 +199,7 @@ void shortlog_add_commit(struct shortlog *log,
> struct commit *commit) if (*buffer)
>  		buffer++;
>  	insert_one_record(log, author, !*buffer ? "<none>" : buffer);
> +	free(key);

Is it the right place for free()?

Here is valgrind output:

==9005== 98,085 bytes in 19,218 blocks are definitely lost in loss record 23 of 25
==9005==    at 0x4A08CFE: malloc (vg_replace_malloc.c:207
==9005==    by 0x4A8D25: xmalloc (wrapper.c:20)
==9005==    by 0x4A8DE1: xmemdupz (wrapper.c:45)
==9005==    by 0x44AFDD: shortlog_add_commit (builtin-shortlog.c:170)
==9005==    by 0x44B31C: cmd_shortlog (builtin-shortlog.c:212)
==9005==    by 0x4049A2: handle_internal_command (git.c:246)
==9005==    by 0x404B63: main (git.c:438)

Re: More 'shortlog' statistics models?

[Jakub Narebski]

Junio C Hamano <gitster@pobox.com> writes:

> Linus Torvalds <torvalds@linux-foundation.org> writes:
> 
> > Anyway, I can do all this with sorting, but it's fun enough that I wonder 
> > if we should make it easier?
> 
> Wasn't one of the GSoC project about git statistics?  Or was it a topic of
> last year?

It was topic of last year GSoC, see http://git.or.cz/gitwiki/SoC2008Projects
It was decided[1] that it would be better for it to remain separate
from git (for support for other SCMs).

  http://repo.or.cz/w/git-stats.git

CC-ed Sverre Rabellier (student for this git-statistic project)

[1]: http://permalink.gmane.org/gmane.comp.version-control.git/90691
-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: More 'shortlog' statistics models?

[Sverre Rabbelier]

Heya,

[message not culled as I suspect Jakub's reply didn't make the git list]

On Wed, Jul 15, 2009 at 06:38, Jakub Narebski<jnareb@gmail.com> wrote:
>> Wasn't one of the GSoC project about git statistics?  Or was it a topic of
>> last year?

That would be me indeed, as Jakub said :).

> It was topic of last year GSoC, see http://git.or.cz/gitwiki/SoC2008Projects
> It was decided[1] that it would be better for it to remain separate
> from git (for support for other SCMs).
>
>  http://repo.or.cz/w/git-stats.git
>
> CC-ed Sverre Rabellier (student for this git-statistic project)
>
> [1]: http://permalink.gmane.org/gmane.comp.version-control.git/90691
> --
> Jakub Narebski
> Poland
> ShadeHawk on #git

Not sure what happened but you addressed me (and my old address at
that) and the git list in one line, not seperated by a comma. Thanks
for the plug regardless :).

-- 
Cheers,

Sverre Rabbelier