Re: Commit ID in exported Tar Ball

["René Scharfe" <rene.scharfe@lsrfire.ath.cx>]

[I'm quoting myself in full because I somehow sent my reply to everyone
but Shawn.  A patch can be found at the end.]

René Scharfe schrieb:
> Shawn O. Pearce schrieb:
>> Ren?? Scharfe <rene.scharfe@lsrfire.ath.cx> wrote:
>>> Shawn O. Pearce schrieb:
>>>> git-describe is more human-friendly than a SHA-1...
>>> Yes, and the Makefile does even more than that: it adds a version
>>> file, a spec file and another version file for git-gui.
>>> 
>>> The first two are probably useful for most projects that actually
>>> do versioned releases.  We could have a simple parser that reads
>>> a template, replaces @@VERSION@@ with a git-describe output
>>> string and adds the result as a synthetic file to the archive.
>>> It's not exactly trivial -- e.g., how to specify git-describe
>>> options, template file and synthetic name, all in one command
>>> line parameter? -- but it's doable.
>> Maybe something just as simple as allowing the user to specify a 
>> shell script in-tree that we unpack and run for them?  That script 
>> prints to stdout the content of the file to include.
> 
> I doubt executing a shell script is simple. :-D  You'd possibly get 
> different results on different platforms (dare I mention Windows?).
> 
> The template system I mentioned would be a kind of scripting language
>  itself, but in this case we define its syntax and can guarantee 
> consistency everywhere git runs.  And since it would only have four 
> types of tokens (@@VERSION@@, @@COMMITID@@, @@@@ and string literals)
> it could be fast and simple.
> 
> We could implement it as a checkout converter, preferably one that is
> only applied by git-archive.  Then we'd rename git.spec.in to
> git.spec, assign the "specfile" attribute to it and let git-archive
> replace the string @@VERSION@@ with git-describe's output.
> git-checkout would not expand the special strings, so you can simply
> edit and version the file as you can do with git.spec.in now.
> Michael would have a file containing only @@COMMITID@@ to solve his
> original problem.  Make sense?
> 
>> So now we're also really talking about, what should git-archive do
>> for a subproject?  Sometimes you really do want to repackage and
>> redistribute the subproject as part of the superproject's tarball.
>> Sometimes you don't.  I think in the case of git.git and 
>> git-gui.git we want to include the subproject.  ;-)
> 
> Oh, yes, subprojects.  git-archive currently exports them as empty 
> directories.  Using tar's append command you could simply build the 
> project+subproject archive in the Makefile.  That wouldn't work well 
> with gitweb, though.  Perhaps a --include-subproject=<path> option is
>  needed?

OK, so here's a first shot at the mentioned parser.  It only understands
@@COMMITID@@ and @@@@, but it's easily extendible.  The internals of
git-describe would need to be converted to library functions, preferably
offering every piece of version info separately (see thread "[PATCH]
Make sure an autogenerated version has at least four parts" for why).

Before doing that, we should determine if this is the way to, though.

René


 Documentation/gitattributes.txt |   19 ++++++-
 archive-tar.c                   |    5 ++-
 archive-zip.c                   |    5 ++-
 cache.h                         |    1 +
 convert.c                       |  112 +++++++++++++++++++++++++++++++++++++++
 5 files changed, 139 insertions(+), 3 deletions(-)

diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
index d3ac9c7..84c414c 100644
--- a/Documentation/gitattributes.txt
+++ b/Documentation/gitattributes.txt
@@ -72,7 +72,7 @@ EFFECTS
 -------
 
 Certain operations by git can be influenced by assigning
-particular attributes to a path.  Currently, three operations
+particular attributes to a path.  Currently, four operations
 are attributes-aware.
 
 Checking-out and checking-in
@@ -374,6 +374,23 @@ frotz	unspecified
 ----------------------------------------------------------------
 
 
+Creating an archive
+~~~~~~~~~~~~~~~~~~~
+
+
+`specfile`
+^^^^^^^^^^
+
+If the attribute `specfile` is set for a file then git will expand
+several placeholders when adding this file to an archive.  The
+expansion depends on the availability of a commit ID, i.e. if
+`git-archive` has been given a tree instead of a commit or a tag
+then no replacement will be done.
+
+`@@COMMITID@@`:: is replaced by the commit hash.
+`@@@@`:: is replaced by `@@`.
+
+
 GIT
 ---
 Part of the gitlink:git[7] suite
diff --git a/archive-tar.c b/archive-tar.c
index 66fe3e3..eba24cb 100644
--- a/archive-tar.c
+++ b/archive-tar.c
@@ -17,6 +17,7 @@ static unsigned long offset;
 static time_t archive_time;
 static int tar_umask = 002;
 static int verbose;
+static const unsigned char *commit_sha1;
 
 /* writes out the whole block, but only if it is full */
 static void write_if_needed(void)
@@ -285,7 +286,8 @@ static int write_tar_entry(const unsigned char *sha1,
 		buffer = NULL;
 		size = 0;
 	} else {
-		buffer = convert_sha1_file(path.buf, sha1, mode, &type, &size);
+		buffer = sha1_file_to_archive(path.buf, sha1, mode, &type,
+		                              &size, commit_sha1);
 		if (!buffer)
 			die("cannot read %s", sha1_to_hex(sha1));
 	}
@@ -304,6 +306,7 @@ int write_tar_archive(struct archiver_args *args)
 
 	archive_time = args->time;
 	verbose = args->verbose;
+	commit_sha1 = args->commit_sha1;
 
 	if (args->commit_sha1)
 		write_global_extended_header(args->commit_sha1);
diff --git a/archive-zip.c b/archive-zip.c
index 444e162..93a5ab3 100644
--- a/archive-zip.c
+++ b/archive-zip.c
@@ -12,6 +12,7 @@
 static int verbose;
 static int zip_date;
 static int zip_time;
+static const unsigned char *commit_sha1;
 
 static unsigned char *zip_dir;
 static unsigned int zip_dir_size;
@@ -195,7 +196,8 @@ static int write_zip_entry(const unsigned char *sha1,
 		if (S_ISREG(mode) && zlib_compression_level != 0)
 			method = 8;
 		result = 0;
-		buffer = convert_sha1_file(path, sha1, mode, &type, &size);
+		buffer = sha1_file_to_archive(path, sha1, mode, &type, &size,
+		                              commit_sha1);
 		if (!buffer)
 			die("cannot read %s", sha1_to_hex(sha1));
 		crc = crc32(crc, buffer, size);
@@ -316,6 +318,7 @@ int write_zip_archive(struct archiver_args *args)
 	zip_dir = xmalloc(ZIP_DIRECTORY_MIN_SIZE);
 	zip_dir_size = ZIP_DIRECTORY_MIN_SIZE;
 	verbose = args->verbose;
+	commit_sha1 = args->commit_sha1;
 
 	if (args->base && plen > 0 && args->base[plen - 1] == '/') {
 		char *base = xstrdup(args->base);
diff --git a/cache.h b/cache.h
index cd875bc..0484904 100644
--- a/cache.h
+++ b/cache.h
@@ -550,6 +550,7 @@ extern void trace_argv_printf(const char **argv, int count, const char *format,
 extern char *convert_to_git(const char *path, const char *src, unsigned long *sizep);
 extern char *convert_to_working_tree(const char *path, const char *src, unsigned long *sizep);
 extern void *convert_sha1_file(const char *path, const unsigned char *sha1, unsigned int mode, enum object_type *type, unsigned long *size);
+extern void *sha1_file_to_archive(const char *path, const unsigned char *sha1, unsigned int mode, enum object_type *type, unsigned long *size, const unsigned char *commit_sha1);
 
 /* match-trees.c */
 void shift_tree(const unsigned char *, const unsigned char *, unsigned char *, int);
diff --git a/convert.c b/convert.c
index 4b26b1a..1cdaec5 100644
--- a/convert.c
+++ b/convert.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "attr.h"
 #include "run-command.h"
+#include "strbuf.h"
 
 /*
  * convert.c - convert a file when checking it out and checking it in.
@@ -667,3 +668,114 @@ void *convert_sha1_file(const char *path, const unsigned char *sha1,
 	}
 	return buffer;
 }
+
+static void strbuf_append(struct strbuf *sb, const void *s, size_t len)
+{
+	if (sb->alloc < sb->len + len) {
+		sb->alloc = (sb->len + len) * 3 / 2 + 16;
+		sb->buf = xrealloc(sb->buf, sb->alloc);
+	}
+	memcpy(sb->buf + sb->len, s, len);
+	sb->len += len;
+}
+
+static unsigned int match_keyword(const char *data, unsigned int datalen,
+                                  const char *keyword)
+{
+	unsigned int keylen = strlen(keyword);
+	if (keylen > datalen)
+		return 0;
+	if (memcmp(data, keyword, keylen))
+		return 0;
+	return keylen;
+}
+
+static void *convert_to_archive(const char *path, const void *src,
+                                unsigned long *sizep,
+                                const unsigned char *commit_sha1)
+{
+	static struct git_attr *attr_specfile;
+	struct git_attr_check check[1];
+	const char *p = src;
+	unsigned long srcsize = *sizep;
+	int at_signs = 0;
+	struct strbuf dst;
+	unsigned int match;
+	int replaced_something = 0;
+
+	if (!commit_sha1)
+		return NULL;
+
+        if (!attr_specfile)
+                attr_specfile = git_attr("specfile", 8);
+
+	check[0].attr = attr_specfile;
+	if (git_checkattr(path, ARRAY_SIZE(check), check))
+		return NULL;
+	if (!ATTR_TRUE(check[0].value))
+		return NULL;
+
+	dst.alloc = srcsize + 128;
+	dst.buf = xmalloc(dst.alloc);
+	dst.len = dst.eof = 0;
+
+	while (srcsize > 0) {
+		if ((at_signs == 0 || at_signs == 1) && *p == '@') {
+			at_signs++;
+			p++;
+			srcsize--;
+			continue;
+		}
+		if (at_signs == 1) {
+			at_signs = 0;
+			strbuf_append(&dst, "@", 1);
+		}
+		if (at_signs == 0) {
+			strbuf_append(&dst, p, 1);
+			p++;
+			srcsize--;
+			continue;
+		}
+
+		if ((match = match_keyword(p, srcsize, "@@")))
+			strbuf_append(&dst, "@@", 2);
+		else if ((match = match_keyword(p, srcsize, "COMMITID@@")))
+			strbuf_append(&dst, sha1_to_hex(commit_sha1), 40);
+		else
+			strbuf_append(&dst, "@@", 2);
+		at_signs = 0;
+		p += match;
+		srcsize -= match;
+		replaced_something = 1;
+	}
+
+	if (!replaced_something) {
+		free(dst.buf);
+		return NULL;
+	}
+
+	*sizep = dst.len;
+	return dst.buf;
+}
+
+void *sha1_file_to_archive(const char *path, const unsigned char *sha1,
+                           unsigned int mode, enum object_type *type,
+                           unsigned long *size,
+                           const unsigned char *commit_sha1)
+{
+	void *buffer = read_sha1_file(sha1, type, size);
+	if (S_ISREG(mode) && buffer) {
+		void *converted = convert_to_working_tree(path, buffer, size);
+		if (converted) {
+			free(buffer);
+			buffer = converted;
+		}
+
+		converted = convert_to_archive(path, buffer, size, commit_sha1);
+		if (converted) {
+			free(buffer);
+			buffer = converted;
+		}
+	}
+	return buffer;
+}