Git archiving only branch work

Git archiving only branch work

[Graeme Geldenhuys]

Hi,

I've strung together the following git alias command for our company. 
This command allows us to create a deployment package (archive) for a 
specific feature. The archive will contain only the files that changed 
during the development of that feature. We then deploy that archive to 
our client's web/database server for example.

[alias]
    deploy = !sh -c 'git archive --prefix=$1/ -o deploy_$1.zip HEAD 
$(git diff --name-only -D $2)' -


usage:
   git deploy OPS123 develop..ops-123


'OPS123' is the prefix directory to store the changes in
'ops-123' is the feature branch the work was done in.

This works very well. The only problem we have so far is that if we 
have files with spaces in the name (eg: SQL update scripts), then the 
command breaks.

Does anybody have an idea on how this can be resolved?  Any help would 
be much appreciated.

Not sure if this is useful, but we are working on Windows systems and 
use Git Bash consoles.

Regards,
  Graeme

Re: Git archiving only branch work

[Peter Krefting]

Graeme Geldenhuys:

> This works very well. The only problem we have so far is that if we have 
> files with spaces in the name (eg: SQL update scripts), then the command 
> breaks.

If you add -z to the git diff command-line, it will give you the names 
with nul terminators instead. If you couple that with xargs -0, you 
should be able to do something like this (untested):

deply = !sh -c 'git diff --name-only -z -D $2 | xargs -x -0 git archive --prefix=$1/ -o 
deploy_$1.zip HEAD'

-- 
\\// Peter - http://www.softwolves.pp.se/

Re: Git archiving only branch work

[Duy Nguyen]

On Thu, Nov 13, 2014 at 12:32:40PM +0000, Graeme Geldenhuys wrote:
> [alias]
>     deploy = !sh -c 'git archive --prefix=$1/ -o deploy_$1.zip HEAD 
> $(git diff --name-only -D $2)' -
> 
> This works very well. The only problem we have so far is that if we 
> have files with spaces in the name (eg: SQL update scripts), then the 
> command breaks.
> 
> Does anybody have an idea on how this can be resolved?  Any help would 
> be much appreciated.

I wonder if it's overkill to do something like this patch ("git
archive" may need some more updates for it to work though). With it
you can do:

  git diff --name-only ... | git archive ... HEAD -- ":(file)-"

The good thing is it works for other commands as well. But is it
really a good thing..

-- 8<--
Subject: [PATCH] pathspec: support :(file)

This pathspec magic must be used alone. It reads the actual pathspec
from a given file whose path is specified after :(file). E.g.

  git ls-files :(file)foo

list files specified by pathspec in file "foo". Reading from stdin is
possible to:

  git ls-fiels :(file)-

TODO: specify line terminator..
---
 pathspec.c | 38 ++++++++++++++++++++++++++++++++++++++
 pathspec.h |  4 +++-
 2 files changed, 41 insertions(+), 1 deletion(-)

diff --git a/pathspec.c b/pathspec.c
index 9304ee3..ba34f9b 100644
--- a/pathspec.c
+++ b/pathspec.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "dir.h"
 #include "pathspec.h"
+#include "argv-array.h"
 
 /*
  * Finds which of the given pathspecs match items in the index.
@@ -72,6 +73,7 @@ static struct pathspec_magic {
 	{ PATHSPEC_GLOB,   '\0', "glob" },
 	{ PATHSPEC_ICASE,  '\0', "icase" },
 	{ PATHSPEC_EXCLUDE, '!', "exclude" },
+	{ PATHSPEC_FROMFILE, 0, "file" },
 };
 
 static void prefix_short_magic(struct strbuf *sb, int prefixlen,
@@ -235,6 +237,9 @@ static unsigned prefix_pathspec(struct pathspec_item *item,
 	} else if (magic & PATHSPEC_FROMTOP) {
 		match = xstrdup(copyfrom);
 		prefixlen = 0;
+	} else if ((magic & PATHSPEC_FROMFILE) && !strcmp(copyfrom, "-")) {
+		match = xstrdup(copyfrom);
+		prefixlen = 0;
 	} else {
 		match = prefix_path_gently(prefix, prefixlen, &prefixlen, copyfrom);
 		if (!match)
@@ -354,6 +359,31 @@ static void NORETURN unsupported_magic(const char *pattern,
 	    pattern, sb.buf);
 }
 
+static void pathspec_fromfile(struct pathspec *pathspec,
+			      unsigned magic_mask, unsigned flags,
+			      const char *prefix, const char *path)
+{
+	struct strbuf buf, nbuf;
+	int line_termination = '\n'; /* FIXME: support :(file:<line terminator>) */
+	struct argv_array av = ARGV_ARRAY_INIT;
+	FILE *fp = !strcmp(path, "-") ? stdin : fopen(path, "r");
+
+	if (!fp)
+		die_errno(_("fail to open %s"), path);
+
+	strbuf_init(&buf, 0);
+	strbuf_init(&nbuf, 0);
+	while (strbuf_getline(&buf, fp, line_termination) != EOF)
+		argv_array_push(&av, buf.buf);
+	strbuf_release(&buf);
+	strbuf_release(&nbuf);
+	if (fp != stdin)
+		fclose(fp);
+	parse_pathspec(pathspec, magic_mask | PATHSPEC_FROMFILE,
+		       flags, prefix, av.argv);
+	/* cannot free av because pathspec keeps references to it */
+}
+
 /*
  * Given command line arguments and a prefix, convert the input to
  * pathspec. die() if any magic in magic_mask is used.
@@ -427,6 +457,14 @@ void parse_pathspec(struct pathspec *pathspec,
 					  item[i].magic & magic_mask,
 					  short_magic);
 
+		if (item[i].magic & PATHSPEC_FROMFILE) {
+			if (n != 1)
+				die(_(":(file) can only be used alone"));
+			pathspec_fromfile(pathspec,
+					  magic_mask | PATHSPEC_FROMFILE,
+					  flags, prefix, item[i].match);
+			return;
+		}
 		if ((flags & PATHSPEC_SYMLINK_LEADING_PATH) &&
 		    has_symlink_leading_path(item[i].match, item[i].len)) {
 			die(_("pathspec '%s' is beyond a symbolic link"), entry);
diff --git a/pathspec.h b/pathspec.h
index 0c11262..84de102 100644
--- a/pathspec.h
+++ b/pathspec.h
@@ -8,13 +8,15 @@
 #define PATHSPEC_GLOB		(1<<3)
 #define PATHSPEC_ICASE		(1<<4)
 #define PATHSPEC_EXCLUDE	(1<<5)
+#define PATHSPEC_FROMFILE	(1<<6)
 #define PATHSPEC_ALL_MAGIC	  \
 	(PATHSPEC_FROMTOP	| \
 	 PATHSPEC_MAXDEPTH	| \
 	 PATHSPEC_LITERAL	| \
 	 PATHSPEC_GLOB		| \
 	 PATHSPEC_ICASE		| \
-	 PATHSPEC_EXCLUDE)
+	 PATHSPEC_EXCLUDE	| \
+	 PATHSPEC_FROMFILE)
 
 #define PATHSPEC_ONESTAR 1	/* the pathspec pattern satisfies GFNM_ONESTAR */
 
-- 
2.1.0.rc0.78.gc0d8480

-- 8< --

Re: Git archiving only branch work

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

> On Thu, Nov 13, 2014 at 12:32:40PM +0000, Graeme Geldenhuys wrote:
>> [alias]
>>     deploy = !sh -c 'git archive --prefix=$1/ -o deploy_$1.zip HEAD 
>> $(git diff --name-only -D $2)' -
>> 
>> This works very well. The only problem we have so far is that if we 
>> have files with spaces in the name (eg: SQL update scripts), then the 
>> command breaks.
>> 
>> Does anybody have an idea on how this can be resolved?  Any help would 
>> be much appreciated.

Set $IFS to newline, so that $(git diff --name-only ...) output is
split at record boundaries, not inside pathnames?

A quick experiment you can do to convince yourself may be:

-- >8 --
#!/bin/sh

data () {
	echo "a"
	echo "b c" ;# SP in between
	echo "d	e " ;# HT and trailing SP
}

show () {
	for i
	do
		echo "<<$i>>"
	done
}

echo ONE
show $(data)

IFS='
'

echo TWO
show $(data)
-- 8< --

On the "git archive" invocation, there may be something that tells
the pathspecs are literal, like '--literal-pathspecs' option, to
avoid metacharacters from being expanded, though.

Re: Git archiving only branch work

[Jeff King]

On Thu, Nov 13, 2014 at 08:36:16PM +0700, Duy Nguyen wrote:

> On Thu, Nov 13, 2014 at 12:32:40PM +0000, Graeme Geldenhuys wrote:
> > [alias]
> >     deploy = !sh -c 'git archive --prefix=$1/ -o deploy_$1.zip HEAD 
> > $(git diff --name-only -D $2)' -
> > 
> > This works very well. The only problem we have so far is that if we 
> > have files with spaces in the name (eg: SQL update scripts), then the 
> > command breaks.
> > 
> > Does anybody have an idea on how this can be resolved?  Any help would 
> > be much appreciated.
> 
> I wonder if it's overkill to do something like this patch ("git
> archive" may need some more updates for it to work though). With it
> you can do:
> 
>   git diff --name-only ... | git archive ... HEAD -- ":(file)-"
> 
> The good thing is it works for other commands as well. But is it
> really a good thing..

I like the idea of taking paths from stdin (and especially if there is a
"-z" option). But using a pathspec that reads from stdin seems like it
creates a lot of corner cases. What would:

  git rev-list --stdin -- ":(file)-"

do? It is kind of neat that you could read from multiple files (besides
stdin), but I'm not sure it is all that useful in practice (you can
always cat them to its stdin).

How about just adding --stdin, which matches other git commands?

-Peff

Re: Git archiving only branch work

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Thu, Nov 13, 2014 at 08:36:16PM +0700, Duy Nguyen wrote:
>
>> On Thu, Nov 13, 2014 at 12:32:40PM +0000, Graeme Geldenhuys wrote:
>> > [alias]
>> >     deploy = !sh -c 'git archive --prefix=$1/ -o deploy_$1.zip HEAD 
>> > $(git diff --name-only -D $2)' -
>> > 
>> > This works very well. The only problem we have so far is that if we 
>> > have files with spaces in the name (eg: SQL update scripts), then the 
>> > command breaks.
>> > 
>> > Does anybody have an idea on how this can be resolved?  Any help would 
>> > be much appreciated.
>> 
>> I wonder if it's overkill to do something like this patch ("git
>> archive" may need some more updates for it to work though). With it
>> you can do:
>> 
>>   git diff --name-only ... | git archive ... HEAD -- ":(file)-"
>> 
>> The good thing is it works for other commands as well. But is it
>> really a good thing..
>
> I like the idea of taking paths from stdin (and especially if there is a
> "-z" option). But using a pathspec that reads from stdin seems like it
> creates a lot of corner cases. What would:
>
>   git rev-list --stdin -- ":(file)-"
>
> do? It is kind of neat that you could read from multiple files (besides
> stdin), but I'm not sure it is all that useful in practice (you can
> always cat them to its stdin).
>
> How about just adding --stdin, which matches other git commands?

How about doing nothing and use the correct $IFS instead?

Re: Git archiving only branch work

[Jeff King]

On Thu, Nov 13, 2014 at 01:10:17PM -0800, Junio C Hamano wrote:

> > How about just adding --stdin, which matches other git commands?
> 
> How about doing nothing and use the correct $IFS instead?

Can you cover all cases with $IFS, including filenames with newlines?

I agree it is probably OK in practice and for the OP's question, but it
is nice to have "-z" variants so you do not have to worry about quoting
at all. I'd argue that a "--stdin -z" should probably also accept raw
filenames, not pathspecs, too (so you do not have to use
"--literal-pathspecs" elsewhere).

-Peff

Re: Git archiving only branch work

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Thu, Nov 13, 2014 at 01:10:17PM -0800, Junio C Hamano wrote:
>
>> > How about just adding --stdin, which matches other git commands?
>> 
>> How about doing nothing and use the correct $IFS instead?
>
> Can you cover all cases with $IFS, including filenames with newlines?

You didn't say "--stdin -z", so I presume --stdin is not solving
anything ;-)

> I agree it is probably OK in practice and for the OP's question, but it
> is nice to have "-z" variants so you do not have to worry about quoting
> at all. I'd argue that a "--stdin -z" should probably also accept raw
> filenames, not pathspecs, too (so you do not have to use
> "--literal-pathspecs" elsewhere).

I agree "--stdin -z" is a good thing but what makes you think that
the producer of the data is _always_ walking the directory hierarchy
and showing the pathnames it sees?  I think use of literal-pathspecs
should not be tied to the use of either --stdin or -z.

Re: Git archiving only branch work

[Jeff King]

On Thu, Nov 13, 2014 at 01:36:48PM -0800, Junio C Hamano wrote:

> > I agree it is probably OK in practice and for the OP's question, but it
> > is nice to have "-z" variants so you do not have to worry about quoting
> > at all. I'd argue that a "--stdin -z" should probably also accept raw
> > filenames, not pathspecs, too (so you do not have to use
> > "--literal-pathspecs" elsewhere).
> 
> I agree "--stdin -z" is a good thing but what makes you think that
> the producer of the data is _always_ walking the directory hierarchy
> and showing the pathnames it sees?  I think use of literal-pathspecs
> should not be tied to the use of either --stdin or -z.

I agree they are technically orthogonal, but I cannot think of a case
where I have ever generated actual _pathspecs_, which might have
wildcards, and needed to use "-z". The point of using "-z" is that you
do not know what crap you are feeding.

Normally I'm in favor of keeping things as flexible as possible, but it
seems very likely that somebody would forget pathspecs in such a case
(the OP did in his example, and I know I have many times in the past).
I don't feel too strongly about it, though.

-Peff

Re: Git archiving only branch work

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> I agree they are technically orthogonal, but I cannot think of a case
> where I have ever generated actual _pathspecs_, which might have
> wildcards, and needed to use "-z". The point of using "-z" is that you
> do not know what crap you are feeding.

You do not have to generate, i.e. you should be allowed to do this:

    $ git cmd --stdin -z <list-of-patterns

And this is not about "flexibility".  Unless your plan is to forbid
a corner case you do not anticipate and always disable pathspec
globbing, you would need to say something like:

	--literal-pathspecs::

        	All Git command lines take dashed options first and
		then revs and then "pathspecs".  They are usually
		used to select the paths using glob(1)-like
		matching, but with this option they must match the
		paths byte-for-byte.

                Except when "--stdin -z" is used, in which case you
                need to give "--no-literal-pathspecs" if you want to
                feed patterns.

Which is awkward.  And "--stdin -z" is most likely used in scripts;
we are not forcing people to keep typing --literal-pathspecs by
leaving them orthogonal *and* people do not have to remember one
more exception (the default of --literal-pathspecs is flipped only
when --stdin -z is in use) to the rule.

Re: Git archiving only branch work

[Jeff King]

On Thu, Nov 13, 2014 at 01:48:12PM -0800, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > I agree they are technically orthogonal, but I cannot think of a case
> > where I have ever generated actual _pathspecs_, which might have
> > wildcards, and needed to use "-z". The point of using "-z" is that you
> > do not know what crap you are feeding.
> 
> You do not have to generate, i.e. you should be allowed to do this:
> 
>     $ git cmd --stdin -z <list-of-patterns

Right. My point is that I am not sure anybody ever really _wants_ to do
this, versus:

  git cmd -- "$pattern1" "$pattern2"

Because patterns tend to be small in number and made with predictable
characters known to the script writer. It is sets of arbitrary filenames
that tend to be long and contain random junk.

> And this is not about "flexibility".  Unless your plan is to forbid
> a corner case you do not anticipate and always disable pathspec
> globbing, you would need to say something like:

I had just assumed we would forbid, but yeah, you could have a switch to
handle either case. That is much nicer to the corner case people.

> Which is awkward.  And "--stdin -z" is most likely used in scripts;
> we are not forcing people to keep typing --literal-pathspecs by
> leaving them orthogonal *and* people do not have to remember one
> more exception (the default of --literal-pathspecs is flipped only
> when --stdin -z is in use) to the rule.

It is not about "forcing to type". It is about "did not realize this was
a potential pitfall and did not write it in the script in the first
place".

-Peff

Re: Git archiving only branch work

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Thu, Nov 13, 2014 at 01:48:12PM -0800, Junio C Hamano wrote:
>
>> Jeff King <peff@peff.net> writes:
>> 
>> > I agree they are technically orthogonal, but I cannot think of a case
>> > where I have ever generated actual _pathspecs_, which might have
>> > wildcards, and needed to use "-z". The point of using "-z" is that you
>> > do not know what crap you are feeding.
>> 
>> You do not have to generate, i.e. you should be allowed to do this:
>> 
>>     $ git cmd --stdin -z <list-of-patterns
>
> Right. My point is that I am not sure anybody ever really _wants_ to do
> this, versus:
>
>   git cmd -- "$pattern1" "$pattern2"
>
> Because patterns tend to be small in number and made with predictable
> characters known to the script writer. It is sets of arbitrary filenames
> that tend to be long and contain random junk.

Perhaps "<filename" may have what made it confusing, as it made it
look as if the script writer has control over it (e.g. configuration
file).  The point actually was that the script invoking --stdin may
not even _know_ the number or the nature of patterns (e.g. end user
input).  Imagine a back-end that receives an RPC request from a
gitweb like front-end that lets you pick a tree and a set of
optional pathspecs, and continue below.

>> And this is not about "flexibility".  Unless your plan is to forbid
>> a corner case you do not anticipate and always disable pathspec
>> globbing, you would need to say something like:
>
> I had just assumed we would forbid,

That design is perfectly fine by me, actually.

I somehow hoped that "--stdin" is (uniformly across subcommands) a
mechanism to let us throw the remainder of what we would have liked
to place on the command line at the command but we couldn't
(e.g. because we feared that too many of them might overflow the
command line length limit) from the standard input stream instead.
As long as you are feeding pathspecs, we should uniformly treat them
as pathspecs no matter where they come from, either from the command
line or the standard input stream.  Special casing "--stdin" with or
without "-z" did not make sense to me in that mindset.

But existing use of "--stdin" is not necessarily "you can feed what
you would otherwise place on command line".  "hash-object --stdin"
is not about reading the name of path that contains the data to be
hashed (the option is "--stdin-path" there).  "update-index --stdin"
is sort of like that but different, in that the underlying command
does not take pathspec in the first place and it takes a "list of
paths" the same way from it takes them from the command line, which
makes it a bad comparison.

So I think "git archive --stdin [-z]" that takes list of paths, not
pathspec, is perfectly fine.  

Re: Git archiving only branch work

[Thomas Koch]

If your servers run a Unix and you can install Git on the servers than you 
might want to try the install script we use in our company:

https://github.com/comsolit/comsolit_deploy

There's a bare git repository on the server and a post-receive hook that 
exports the content of the git repository in a predefined folder. After that a 
symlink is switched to the new version.

You can run hook scripts after the export (checkout) and after the switch.

The script lacks documentation... (PRs welcome!) But it is unit tested!

Regards, Thomas Koch