Git development
 help / color / mirror / Atom feed
* What's a good setup for submitting patches to the list properly?
From: Thell Fowler @ 2009-08-20  0:09 UTC (permalink / raw)
  To: git

Sorry for once again.

I haven't been able to figure out a good setup for posting patches to the 
list correctly, perhaps someone could tell me where I'm going wrong.

Alpine is setup to access git @ tbfowler.name with the 'postpone' folder 
being the 'Drafts' folder on the remote mail host, and a local mbox folder 
~/mail/git

Locally I prepped the emails using:

git format-patch --cover-letter --full-index -n 
--in-reply-to=1249428804.2774.52.camel@GWPortableVCS --thread --signoff -6 
--stdout>>~/mail/git

http://article.gmane.org/gmane.comp.version-control.git/124834

Thinking that the cover letter would be in reply to a previous thread, and 
that the rest would show as a reply to that.  After doing the 
format-patch, I went into Alpine's git folder selected the messages and 
saved them to the Drafts folder, then did 'compose' for each one, filling 
in the information I thought was needed.

At that point, the msg list looked flat, so I opened Evolution Mail and 
looked at them there, they cascaded properly, so I sent them.  Yet, now, 
looking on gmane each of the patch msgs is a top level post and the cover 
letter correctly posted as a reply to the previous thread.

What could I have done/checked before sending to make sure that these 
would have posted properly?

-- 
Thell

^ permalink raw reply

* [PATCH 6/6] Add diff tests for trailing-space on incomplete lines
From: Thell Fowler @ 2009-08-19 23:09 UTC (permalink / raw)
  To: Johannes Schindelin; +Cc: git
In-Reply-To: <cover.1250719760.git.git@tbfowler.name>

  - Adds 7 --no-index tests to t4015-diff-whitespace.sh specifically
    to ensure that xutils.c xdl_hash_record_with_whitespace and
    xdl_recmatch process to the end of the record and handle an
    incomplete line terminator the same as a new-line.

Signed-off-by: Thell Fowler <git@tbfowler.name>
---
 t/t4015-diff-whitespace.sh |   33 +++++++++++++++++++++++++++++++++
 1 files changed, 33 insertions(+), 0 deletions(-)

diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh
index 6d13da30dad5a78fb17a01e86ef33072ea9e6250..193ddbe0659ede17154ffda3b25ebc5e6c686d6e 100755
--- a/t/t4015-diff-whitespace.sh
+++ b/t/t4015-diff-whitespace.sh
@@ -395,4 +395,37 @@ test_expect_success 'combined diff with autocrlf conversion' '
 
 '
 
+# Ignore trailing-space testing on incomplete lines.
+prepare_diff_file () {
+	printf "%s%$2s" foo "" >"$1"
+	if [ $3 = "+nl" ]
+	then
+		printf "\n" >>"$1"
+	fi
+}
+
+diff_trailing () {
+	foo="foo___"
+	prepare_diff_file "left" "$2" "$3"
+	lfoo=$( expr substr $foo 1 $((3+$2)) )
+	lfoo=${lfoo}"$3"
+
+	prepare_diff_file "right" "$4" "$5"
+	rfoo=$( expr substr $foo 1 $((3+$4)) )
+	rfoo=${rfoo}"$5"
+
+	label="-$1 $lfoo $rfoo"
+
+	test_expect_success "$label" \
+	"! git diff --no-index -$1 -- left right | grep -q foo"
+}
+
+diff_trailing w 0 +nl 1 -nl
+diff_trailing w 0 -nl 1 -nl
+diff_trailing b 0 +nl 0 -nl
+diff_trailing b 1 +nl 0 -nl
+diff_trailing b 1 -nl 0 -nl
+diff_trailing -ignore-space-at-eol 0 +nl 0 -nl
+diff_trailing -ignore-space-at-eol 2 +nl 2 -nl
+
 test_done
-- 
1.6.4.172.g5c0d0.dirty

^ permalink raw reply related

* [PATCH 5/6] Make diff --ignore-space-at-eol handle incomplete lines.
From: Thell Fowler @ 2009-08-19 23:08 UTC (permalink / raw)
  To: Johannes Schindelin; +Cc: git
In-Reply-To: <cover.1250719760.git.git@tbfowler.name>

  - When processing with --ignore-space-change a diff would be found
    whenever an incomplete line was encountered.  xdl_recmatch should
    process the full length of the record instead of assuming both
    sides have a terminator.

Signed-off-by: Thell Fowler <git@tbfowler.name>
---
 xdiff/xutils.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/xdiff/xutils.c b/xdiff/xutils.c
index e126de450c99fb1e557c2cfc0ffe54e8e3e80394..a8ed102d528bdb5d8f0839eb392b35dc1c534fba 100644
--- a/xdiff/xutils.c
+++ b/xdiff/xutils.c
@@ -216,7 +216,7 @@ int xdl_recmatch(const char *l1, long s1, const char *l2, long s2, long flags)
 		}
 		return (i1 >= s1 && i2 >= s2);
 	} else if (flags & XDF_IGNORE_WHITESPACE_AT_EOL) {
-		for (i1 = i2 = 0; i1 < s1 && i2 < s2; ) {
+		for (i1 = i2 = 0; i1 <= s1 && i2 <= s2; ) {
 			if (l1[i1] != l2[i2]) {
 				while (i1 < s1 && isspace(l1[i1]))
 					i1++;
-- 
1.6.4.172.g5c0d0.dirty

^ permalink raw reply related

* [PATCH 4/6] Make diff -b handle trailing-spaces on incomplete lines.
From: Thell Fowler @ 2009-08-19 23:07 UTC (permalink / raw)
  To: Johannes Schindelin; +Cc: git
In-Reply-To: <cover.1250719760.git.git@tbfowler.name>

  - When processing trailing spaces with --ignore-space-change a diff
    would be found whenever an incomplete line terminated before the
    whitespace handling started regardless of actual trailing-spaces.
    xdl_recmatch should process the full length of the record instead
    of assuming both sides have a terminator, and should treat the
    terminator as a whitespace like it does with '\n'.

Signed-off-by: Thell Fowler <git@tbfowler.name>
---
 xdiff/xutils.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/xdiff/xutils.c b/xdiff/xutils.c
index 1f28f4fb4e0a8fdc6c9aa1904cf0362dd1e7b977..e126de450c99fb1e557c2cfc0ffe54e8e3e80394 100644
--- a/xdiff/xutils.c
+++ b/xdiff/xutils.c
@@ -203,9 +203,9 @@ int xdl_recmatch(const char *l1, long s1, const char *l2, long s2, long flags)
 		}
 		return (i1 >= s1 && i2 >= s2);
 	} else if (flags & XDF_IGNORE_WHITESPACE_CHANGE) {
-		for (i1 = i2 = 0; i1 < s1 && i2 < s2; ) {
-			if (isspace(l1[i1])) {
-				if (!isspace(l2[i2]))
+		for (i1 = i2 = 0; i1 <= s1 && i2 <= s2; ) {
+			if (isspace(l1[i1]) || (i1 == s1 && i2 < s2)) {
+				if (!isspace(l2[i2]) && i2 != s2)
 					return 0;
 				while (isspace(l1[i1]) && i1 < s1)
 					i1++;
-- 
1.6.4.172.g5c0d0.dirty

^ permalink raw reply related

* [RFC/PATCH 7/6] git clone: Add --recursive to automatically checkout (nested) submodules
From: Johan Herland @ 2009-08-19 23:07 UTC (permalink / raw)
  To: git; +Cc: gitster, johan, barkalow
In-Reply-To: <1250646324-961-1-git-send-email-johan@herland.net>

Many projects using submodules expect all submodules to be checked out
in order to build/work correctly. A common command sequence for
developers on such projects is:

	git clone url/to/project
	cd project
	git submodule update --init (--recursive)

This patch introduces the --recursive option to git-clone. The new
option causes git-clone to recursively clone and checkout all
submodules of the cloned project. Hence, the above command sequence
can be reduced to:

	git clone --recursive url/to/project

--recursive is ignored if no checkout is done by the git-clone.

The patch also includes documentation and a selftest.

Signed-off-by: Johan Herland <johan@herland.net>
---

Hi,

It just hit me today that yesterday's patch series missed the last part
of the puzzle...

I'm not at all married to the '--recursive' name for this option, but
I wasn't able to think of anything better that wasn't too long.


Have fun! :)

...Johan


 Documentation/git-clone.txt  |   10 +++++++++-
 builtin-clone.c              |   11 ++++++++++-
 t/t7407-submodule-foreach.sh |   12 ++++++++++++
 3 files changed, 31 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 2c63a0f..88ea272 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -12,7 +12,7 @@ SYNOPSIS
 'git clone' [--template=<template_directory>]
 	  [-l] [-s] [--no-hardlinks] [-q] [-n] [--bare] [--mirror]
 	  [-o <name>] [-u <upload-pack>] [--reference <repository>]
-	  [--depth <depth>] [--] <repository> [<directory>]
+	  [--depth <depth>] [--recursive] [--] <repository> [<directory>]
 
 DESCRIPTION
 -----------
@@ -147,6 +147,14 @@ objects from the source repository into a pack in the cloned repository.
 	with a long history, and would want to send in fixes
 	as patches.
 
+--recursive::
+	After the clone is created, initialize all submodules within,
+	using their default settings. This is equivalent to running
+	'git submodule update --init --recursive' immediately after
+	the clone is finished. This option is ignored if the cloned
+	repository does not have a worktree/checkout (i.e. if any of
+	`--no-checkout`/`-n`, `--bare`, or `--mirror` is given)
+
 <repository>::
 	The (possibly remote) repository to clone from.  See the
 	<<URLS,URLS>> section below for more information on specifying
diff --git a/builtin-clone.c b/builtin-clone.c
index 32dea74..0d2b4a8 100644
--- a/builtin-clone.c
+++ b/builtin-clone.c
@@ -38,7 +38,7 @@ static const char * const builtin_clone_usage[] = {
 };
 
 static int option_quiet, option_no_checkout, option_bare, option_mirror;
-static int option_local, option_no_hardlinks, option_shared;
+static int option_local, option_no_hardlinks, option_shared, option_recursive;
 static char *option_template, *option_reference, *option_depth;
 static char *option_origin = NULL;
 static char *option_upload_pack = "git-upload-pack";
@@ -59,6 +59,8 @@ static struct option builtin_clone_options[] = {
 		    "don't use local hardlinks, always copy"),
 	OPT_BOOLEAN('s', "shared", &option_shared,
 		    "setup as shared repository"),
+	OPT_BOOLEAN(0, "recursive", &option_recursive,
+		    "setup as shared repository"),
 	OPT_STRING(0, "template", &option_template, "path",
 		   "path the template repository"),
 	OPT_STRING(0, "reference", &option_reference, "repo",
@@ -73,6 +75,10 @@ static struct option builtin_clone_options[] = {
 	OPT_END()
 };
 
+static const char *argv_submodule[] = {
+	"submodule", "update", "--init", "--recursive", NULL
+};
+
 static char *get_repo_path(const char *repo, int *is_bundle)
 {
 	static char *suffix[] = { "/.git", ".git", "" };
@@ -608,6 +614,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 
 		err |= run_hook(NULL, "post-checkout", sha1_to_hex(null_sha1),
 				sha1_to_hex(remote_head->old_sha1), "1", NULL);
+
+		if (!err && option_recursive)
+			err = run_command_v_opt(argv_submodule, RUN_GIT_CMD);
 	}
 
 	strbuf_release(&reflog_msg);
diff --git a/t/t7407-submodule-foreach.sh b/t/t7407-submodule-foreach.sh
index de1730d..25ec281 100755
--- a/t/t7407-submodule-foreach.sh
+++ b/t/t7407-submodule-foreach.sh
@@ -220,4 +220,16 @@ test_expect_success 'test "status --recursive"' '
 	test_cmp expect actual
 '
 
+test_expect_success 'use "git clone --recursive" to checkout all submodules' '
+	git clone --recursive super clone4 &&
+	test -d clone4/.git &&
+	test -d clone4/sub1/.git &&
+	test -d clone4/sub2/.git &&
+	test -d clone4/sub3/.git &&
+	test -d clone4/nested1/.git &&
+	test -d clone4/nested1/nested2/.git &&
+	test -d clone4/nested1/nested2/nested3/.git &&
+	test -d clone4/nested1/nested2/nested3/submodule/.git
+'
+
 test_done
-- 
1.6.4.304.g1365c.dirty

^ permalink raw reply related

* [PATCH 3/6] Make diff -w handle trailing-spaces on incomplete lines.
From: Thell Fowler @ 2009-08-19 23:07 UTC (permalink / raw)
  To: Johannes Schindelin; +Cc: git
In-Reply-To: <cover.1250719760.git.git@tbfowler.name>

  - When processing trailing spaces with --ignore-all-space a diff
    would be found whenever one side had 0 spaces and either (or both)
    sides was an incomplete line.  xdl_recmatch should process the
    full length of the record instead of assuming both sides have a
    terminator.

Signed-off-by: Thell Fowler <git@tbfowler.name>
---
 xdiff/xutils.c |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/xdiff/xutils.c b/xdiff/xutils.c
index c6512a53b08a8c9039614738310aa2786f4fbb1c..1f28f4fb4e0a8fdc6c9aa1904cf0362dd1e7b977 100644
--- a/xdiff/xutils.c
+++ b/xdiff/xutils.c
@@ -191,14 +191,14 @@ int xdl_recmatch(const char *l1, long s1, const char *l2, long s2, long flags)
 	int i1, i2;
 
 	if (flags & XDF_IGNORE_WHITESPACE) {
-		for (i1 = i2 = 0; i1 < s1 && i2 < s2; ) {
+		for (i1 = i2 = 0; i1 <= s1 && i2 <= s2; ) {
 			if (isspace(l1[i1]))
-				while (isspace(l1[i1]) && i1 < s1)
+				while (isspace(l1[i1]) && i1 <= s1)
 					i1++;
 			if (isspace(l2[i2]))
-				while (isspace(l2[i2]) && i2 < s2)
+				while (isspace(l2[i2]) && i2 <= s2)
 					i2++;
-			if (i1 < s1 && i2 < s2 && l1[i1++] != l2[i2++])
+			if (i1 <= s1 && i2 <= s2 && l1[i1++] != l2[i2++])
 				return 0;
 		}
 		return (i1 >= s1 && i2 >= s2);
-- 
1.6.4.172.g5c0d0.dirty

^ permalink raw reply related

* [PATCH 2/6] Make xdl_hash_record_with_whitespace ignore eof
From: Thell Fowler @ 2009-08-19 23:06 UTC (permalink / raw)
  To: Johannes Schindelin; +Cc: git
In-Reply-To: <cover.1250719760.git.git@tbfowler.name>

  - When xdl_hash_record_with_whitespace encountered an incomplete
    line the hash would be different than the identical line with
    either --ignore-space-change or --ignore-space-at-eol on an
    incomplete line because they only terminated with a check for
    a new-line.

Signed-off-by: Thell Fowler <git@tbfowler.name>
---
 xdiff/xutils.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/xdiff/xutils.c b/xdiff/xutils.c
index 04ad468702209b77427e635370d41001986042ce..c6512a53b08a8c9039614738310aa2786f4fbb1c 100644
--- a/xdiff/xutils.c
+++ b/xdiff/xutils.c
@@ -248,12 +248,12 @@ static unsigned long xdl_hash_record_with_whitespace(char const **data,
 			if (flags & XDF_IGNORE_WHITESPACE)
 				; /* already handled */
 			else if (flags & XDF_IGNORE_WHITESPACE_CHANGE
-					&& ptr[1] != '\n') {
+					&& ptr[1] != '\n' && ptr + 1 < top) {
 				ha += (ha << 5);
 				ha ^= (unsigned long) ' ';
 			}
 			else if (flags & XDF_IGNORE_WHITESPACE_AT_EOL
-					&& ptr[1] != '\n') {
+					&& ptr[1] != '\n' && ptr + 1 < top) {
 				while (ptr2 != ptr + 1) {
 					ha += (ha << 5);
 					ha ^= (unsigned long) *ptr2;
-- 
1.6.4.172.g5c0d0.dirty

^ permalink raw reply related

* [PATCH] graph API: display uninteresting commits as '^' instead of '*'
From: Adam Simpkins @ 2009-08-19 23:06 UTC (permalink / raw)
  To: git; +Cc: Junio C Hamano

Using --graph and --show-all together now displays UNINTERESTING commits
using '^' characters instead of '*'.

Something like the following command will demonstrate the change:

   git log --graph --show-all ^HEAD~2 HEAD

Signed-off-by: Adam Simpkins <simpkins@facebook.com>
---
 graph.c |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/graph.c b/graph.c
index 6746d42..50b68a4 100644
--- a/graph.c
+++ b/graph.c
@@ -775,6 +775,14 @@ static void graph_output_commit_char(struct git_graph *graph, struct strbuf *sb)
 	}
 
 	/*
+	 * For UNINTERESTING commits (displayed with --show-all), print '^'
+	 */
+	if (graph->commit->object.flags & UNINTERESTING) {
+		strbuf_addch(sb, '^');
+		return;
+	}
+
+	/*
 	 * If revs->left_right is set, print '<' for commits that
 	 * come from the left side, and '>' for commits from the right
 	 * side.
-- 
1.6.0.4

^ permalink raw reply related

* [PATCH 1/6] Add supplemental test for trailing-whitespace on incomplete lines.
From: Thell Fowler @ 2009-08-19 23:06 UTC (permalink / raw)
  To: Johannes Schindelin; +Cc: git
In-Reply-To: <cover.1250719760.git.git@tbfowler.name>

*** For illustrative purposes only and not meant for upstream ***

  - Adds a stand-alone test that loops through A-side B-side with
    and without new-lines from 0 to 3 spaces per side.
    This is a draft test meant to expose the issue with xutils.c
    handling of incomplete lines and trailing-spaces.

Signed-off-by: Thell Fowler <git@tbfowler.name>
---
 t/t4015-diff-trailing-whitespace.sh |   95 +++++++++++++++++++++++++++++++++++
 1 files changed, 95 insertions(+), 0 deletions(-)
 create mode 100755 t/t4015-diff-trailing-whitespace.sh

diff --git a/t/t4015-diff-trailing-whitespace.sh b/t/t4015-diff-trailing-whitespace.sh
new file mode 100755
index 0000000000000000000000000000000000000000..c4937c1b457c24b35565b09e7b262443a05f9795
--- /dev/null
+++ b/t/t4015-diff-trailing-whitespace.sh
@@ -0,0 +1,95 @@
+#!/bin/sh
+
+test_description='Test trailing whitespace in diff engine.
+
+'
+. ./test-lib.sh
+. "$TEST_DIRECTORY"/diff-lib.sh
+
+# Trailing-space testing with and without newlines.
+prepare_diff_file () {
+	printf "%s%$2s" foo "" >"$1"
+	if [ $3 = "+nl" ]
+	then
+		printf "\n" >>"$1"
+	fi
+}
+
+diff_trailing () {
+	foo="foo___"
+	prepare_diff_file "left" "$2" "$3"
+	lfoo=$( expr substr $foo 1 $((3+$2)) )
+	lfoo=${lfoo}"$3"
+
+	prepare_diff_file "right" "$4" "$5"
+	rfoo=$( expr substr $foo 1 $((3+$4)) )
+	rfoo=${rfoo}"$5"
+
+	label="-$1 $lfoo $rfoo ($6)"
+
+	if [ "$6" != "should_diff" ]
+	then
+		negate='!'
+	else
+		negate=''
+	fi
+
+	if [ -z "$7" ]
+	then
+		test_expect_success "$label" \
+		"$negate git diff --no-index -$1 -- left right | grep -q foo"
+	else
+		test_expect_failure "$label" \
+		"$negate git diff --no-index -$1 -- left right | grep -q foo"
+	fi
+
+	test_debug "git diff --no-index -$1 -- left right | grep foo"
+}
+
+touch diffout
+for arg in -ignore-all-space -ignore-space-at-eol -ignore-space-change
+do
+	for i1 in 0 1 2 3
+	do
+		for i2 in 0 1 2 3
+		do
+			diff_trailing $arg $i1 +nl $i2 -nl should_not_diff >> diffout
+			diff_trailing $arg $i1 -nl $i2 +nl should_not_diff >> diffout
+
+			if [ $i1 -ne $i2 ]
+			then
+				diff_trailing $arg $i1 +nl $i2 +nl should_not_diff >> diffout
+				diff_trailing $arg $i1 -nl $i2 -nl should_not_diff >> diffout
+			fi
+		done
+	done
+done
+
+test_debug 'grep "FAIL" diffout'
+
+for arg in all eol change
+do
+	grep "FAIL" diffout | \
+	grep "$arg" | \
+	cut -d " " -f 4- | \
+
+	##  Playing with filtering to isolate core issue.
+	#sort -k 2,2 -k 3,3 | \
+	#awk '{ forward = $2 " " $3; reverse = $3 " " $2}
+	#	!seen[forward]++ && !seen[reverse]++' | \
+	#sort -k 2,2 | \
+
+	##  Playing with filtering to isolate core issue.
+	##  This seems like the most illustrative output...
+	awk '{ key=$3 ; gsub(/-/, "+", key) ; key=$2 ":" key ; if ( hash[key]++ == 0 ) print ; }'
+	
+	##  Playing with filtering to isolate core issue.
+	#awk '{ if ( $3 ~ /.*\-/ )
+	#		print $0
+	#	else
+	#		print $1 " " $3 " " $2 " " $4
+	#	; }' | \
+	#sort -k 2,2 -k 3,3
+done
+
+test_done
-- 
1.6.4.172.g5c0d0.dirty

^ permalink raw reply related

* [PATCH 0/6 RFC] Series to correct xutils incomplete line handling.
From: Thell Fowler @ 2009-08-19 23:05 UTC (permalink / raw)
  To: Johannes Schindelin; +Cc: git
In-Reply-To: <1249428804.2774.52.camel@GWPortableVCS>


[1/6]
Add supplemental test for trailing-whitespace on incomplete lines.

-- This patch is for illustrative purposes only. It exposes the current 
failures of git diff whitespace ignore options when dealing with trailing-
spaces on incomplete lines.

[2/6] through [5/6]
Make xdl_hash_record_with_whitespace ignore eof
Make diff -w handle trailing-spaces on incomplete lines.
Make diff -b handle trailing-spaces on incomplete lines.
Make diff --ignore-space-at-eol handle incomplete lines.

-- These alter the record ptr loops to go to the end of the record and to 
treat the terminator as it would '\n'.

[6/6]
Add diff tests for trailing-space on incomplete lines

-- Just the seven test cases that would identify future breakage.


 t/t4015-diff-trailing-whitespace.sh |   95 +++++++++++++++++++++++++++++++++++
 t/t4015-diff-whitespace.sh          |   33 ++++++++++++
 xdiff/xutils.c                      |   20 ++++----
 3 files changed, 138 insertions(+), 10 deletions(-)
 create mode 100755 t/t4015-diff-trailing-whitespace.sh

^ permalink raw reply

* [PATCH] Add test case for rev-list --parents --show-all
From: Adam Simpkins @ 2009-08-19 22:58 UTC (permalink / raw)
  To: Junio C Hamano, Santi Béjar, Git Mailing List
In-Reply-To: <20090819225547.GR8147@facebook.com>

This test case ensures that rev-list --parents --show-all gets the
parent history correct.  Normally, --parents rewrites parent history to
skip TREESAME parents.  However, --show-all causes TREESAME parents to
still be included in the revision list, so the parents should still be
included too.

Signed-off-by: Adam Simpkins <simpkins@facebook.com>
---

Looking through the code, I believe TREESAME commits are the only ones
affected by my earlier bug in simplify_commit().

 t/t6015-rev-list-show-all-parents.sh |   31 +++++++++++++++++++++++++++++++
 1 files changed, 31 insertions(+), 0 deletions(-)
 create mode 100644 t/t6015-rev-list-show-all-parents.sh

diff --git a/t/t6015-rev-list-show-all-parents.sh b/t/t6015-rev-list-show-all-parents.sh
new file mode 100644
index 0000000..8b146fb
--- /dev/null
+++ b/t/t6015-rev-list-show-all-parents.sh
@@ -0,0 +1,31 @@
+#!/bin/sh
+
+test_description='--show-all --parents does not rewrite TREESAME commits'
+
+. ./test-lib.sh
+
+test_expect_success 'set up --show-all --parents test' '
+	test_commit one foo.txt &&
+	commit1=`git rev-list -1 HEAD` &&
+	test_commit two bar.txt &&
+	commit2=`git rev-list -1 HEAD` &&
+	test_commit three foo.txt &&
+	commit3=`git rev-list -1 HEAD`
+	'
+
+test_expect_success '--parents rewrites TREESAME parents correctly' '
+	echo $commit3 $commit1 > expected &&
+	echo $commit1 >> expected &&
+	git rev-list --parents HEAD -- foo.txt > actual &&
+	test_cmp expected actual
+	'
+
+test_expect_success '--parents --show-all does not rewrites TREESAME parents' '
+	echo $commit3 $commit2 > expected &&
+	echo $commit2 $commit1 >> expected &&
+	echo $commit1 >> expected &&
+	git rev-list --parents --show-all HEAD -- foo.txt > actual &&
+	test_cmp expected actual
+	'
+
+test_done
-- 
1.6.0.4

^ permalink raw reply related

* Re: [PATCH] graph API: fix bug in graph_is_interesting()
From: Adam Simpkins @ 2009-08-19 22:55 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Santi Béjar, Git Mailing List
In-Reply-To: <7v4os41frm.fsf@alter.siamese.dyndns.org>

On Tue, Aug 18, 2009 at 11:25:49PM -0700, Junio C Hamano wrote:
> Junio C Hamano <gitster@pobox.com> writes:
> >
> > When simplify_commit() logic (now called get_comit_action()) decides to
> > show this commit because revs->show_all was specified, we did not rewrite
> > its parents, but now we will?
> 
> That is, here is what I meant...
> 
> -	if (action == commit_show && revs->prune && revs->dense && want_ancestry(revs)) {
> +	if (action == commit_show &&
> +	    !revs->show_all &&
> +	    revs->prune && revs->dense && want_ancestry(revs)) {
> 
> We may want to add some tests to demonstrate the breakage this fix
> addresses.

Yes, you're right.  Thanks for catching that.  I'll submit a test case
that checks this scenario.

-- 
Adam Simpkins
simpkins@facebook.com

^ permalink raw reply

* Re: Continue git clone after interruption
From: René Scharfe @ 2009-08-19 22:23 UTC (permalink / raw)
  To: Nicolas Pitre; +Cc: Johannes Schindelin, Jakub Narebski, Tomasz Kontusz, git
In-Reply-To: <alpine.LFD.2.00.0908191122020.6044@xanadu.home>

Nicolas Pitre schrieb:
> 3) "git archive --remote=git://foo.bar/baz CLONE_HEAD" and store the 
>    result locally. Keep track of how many files are received, and how 
>    many bytes for the currently received file.
> 
> 4) if network connection is broken, loop back to (3) adding
>    --skip=${nr_files_received},${nr_bytes_in_curr_file_received} to
>    the git-archive argument list.  REmote server simply skips over 
>    specified number of files and bytes into the next file.
> 
> 5) Get content from remote commit object for CLONE_HEAD somehow. (?)

[...]

> - That probably would be a good idea to have a tgz format to 'git
>   archive' which might be simpler to deal with than the zip format.

Adding support for the tgz format would be useful anyway, I guess, and
is easy to implement.

And adding support for cpio (and cpio.gz) and writing an extractor for
it should be simpler than writing a tar extractor alone.

One needs to take a closer look at the limits of the chosen archive
format (file name length, supported file types and attributes, etc.) to
make sure any archive can be turned back into the same git tree.

The commit object could be sent as the first (fake) file of the archive.

You'd need a way to turn off the effect of the attributes export-subst
and export-ignore.

Currently, convert_to_working_tree() is used on the contents of all
files in an archive.  You'd need a way to turn that off, too.

Adding a new format type is probably the easiest way to bundle the
special requirements of the previous three paragraphs.

René

^ permalink raw reply

* Re: question concerning branches
From: Linus Torvalds @ 2009-08-19 21:51 UTC (permalink / raw)
  To: Ingo Brueckl; +Cc: Jakub Narebski, git
In-Reply-To: <4a8c51f5@wupperonline.de>



On Wed, 19 Aug 2009, Ingo Brueckl wrote:

> Jakub Narebski <jnareb@gmail.com> writes:
> 
> > You finish old work (or stash it away), _then_ you begin new work.
> 
> Ok, this helps me a little bit to understand.
> 
> The branches aren't designed to split my work, but rather something to
> collect the different parts of my work.

Hmm. Yes. That's one way of looking at it.

At the same time, thinking about it another way may explain the git 
choices in this area. There's two issues:

 - if we _don't_ carry the edits around across branch switches, then what 
   would we do?

   Basically, since you haven't committed things, it's kind of floating 
   around. You switch to another branch, what should we do? There are 
   really only two choices: either we'd need to 'stash' the state with the 
   branch we switch away from (which is apparently what you expected), or 
   we need to just move the changes to the new branch (which is what git 
   does, or complains if it cannot).

   Now, 'stashing' the changes is actually very much against the whole git 
   philosophy. Git was built up around the index and the database, and 
   branches have always been pointers to the top-of-commit, so there 
   literally isn't any way to stash things that makes sense. Sure, later 
   on we ended up having the 'stash' command, but that's totally separate 
   from branches, and is an independently useful thing.

 - One of the big reasons to act like git does is that the way at least 
   _I_ work is to actually create a new branch with the explicit intention 
   of committing work I have already done!

   IOW, your example was

	git branch test
	git checkout test
	# edit foo.bar
	git checkout master

   and you were surprised that the edit followed you back to the "master" 
   branch, but what is actualyl a much more natural way of working is

	# edit foo.bar
	# realize that this was actually the start of a new feature
	git branch new-feature
	git checkout new-feature
	# maybe continue to edit foo.bar until it's all good
	git commit -a

   ie the git behavior explicitly _encourages_ you to not have to decide 
   before-the-fact to create a branch - it may be that only after you've 
   done the changes do you realize that "oops, these changes were _way_ 
   more intrusive than I originally anticipated, and I don't want to 
   commit them on the master branch, I want to commit them on an 
   experimental topic branch instead"

So there are two different reasons why git works the way it does: a pure 
implementation reason ("working any other way would not fit the git 
model") and a practical workflow reason ("you are _expected_ to move dirty 
state around with your branches, because one common case is to create a 
branch _for_ that dirty state").

			Linus

^ permalink raw reply

* Re: Continue git clone after interruption
From: Nicolas Pitre @ 2009-08-19 21:13 UTC (permalink / raw)
  To: Jakub Narebski; +Cc: Tomasz Kontusz, git, Johannes Schindelin
In-Reply-To: <200908192142.51384.jnareb@gmail.com>

On Wed, 19 Aug 2009, Jakub Narebski wrote:

> Cc-ed Dscho, so he can easier participate in this subthread.
> 
> On Wed, 19 Aug 2009, Nicolas Pitre wrote:
> > On Wed, 19 Aug 2009, Jakub Narebski wrote:
> 
> > > P.S. What do you think about 'bundle' capability extension mentioned
> > >      in a side sub-thread?
> > 
> > I don't like it.  Reason is that it forces the server to be (somewhat) 
> > stateful by having to keep track of those bundles and cycle them, and it 
> > doubles the disk usage by having one copy of the repository in the form 
> > of the original pack(s) and another copy as a bundle.
> 
> I agree about problems with disk usage, but I disagree about server
> having to be stateful; server can just simply scan for bundles, and
> offer links to them if client requests 'bundles' capability, somewhere
> around initial git-ls-remote list of refs.

But that's the client that has to deal with what the server wants to 
offer, instead of the server actually serving data as the client wants.

> Well, offering daily bundle in addition to daily snapshot could be
> a good practice, at least until git acquires resumable fetch (resumable
> clone).

Outside of Git: maybe.  Through the git protocol: no.  And what would 
that bundle contain over the daily snapshot?  The whole history?  If so 
that goes against the idea that people concerned by all this have slow 
links and probably aren't interested in the time to download it all.  If 
the bundle contains only the top revision then it has no advantage over 
the snapshot.  Somewhere in the middle?  Sure, but then where to draw 
the line?  That's for the client to decide, not the server 
administrator.

And what if you start your slow transfer which breaks in the middle.  
The next morning you want to restart it in the hope that you might 
resume the transfer of the bundle that is incomplete.  But crap, the 
server has updated its bundle and your half-bundle is now useless. 
You've wasted your bandwidth for nothing.

> > If you think about git.kernel.org which has maybe hundreds of 
> > repositories where the big majority of them are actually forks of Linus' 
> > own repository, then having all those forks reference Linus' repository 
> > is a big disk space saver (and IO too as the referenced repository is 
> > likely to remain cached in memory).  Having a bundle ready for each of 
> > them will simply kill that space advantage, unless they all share the 
> > same bundle.
> 
> I am thinking about sharing the same bundle for related projects.

... meaning more administrative burden.

> > Now sharing that common bundle could be done of course, but that makes 
> > things yet more complex while still wasting IO because some requests 
> > will hit the common pack and some others will hit the bundle, making 
> > less efficient usage of the disk cache on the server.
> 
> Hmmm... true (unless bundles are on separate server).

... meaning additional but avoidable costs.

> > Yet, that bundle would probably not contain the latest revision if it is 
> > only periodically updated, even less so if it is shared between multiple 
> > repositories as outlined above.  And what people with slow/unreliable 
> > network links are probably most interested in is the latest revision and 
> > maybe a few older revisions, but probably not the whole repository as 
> > that is simply too long to wait for.  Hence having a big bundle is not 
> > flexible either with regards to the actual data transfer size.
> 
> I agree that bundle would be useful for restartable clone, and not
> useful for restartable fetch.  Well, unless you count (non-existing)
> GitTorrent / git-mirror-sync as this solution... ;-)

I don't think fetches after a clone are such an issue.  They are 
typically transfers being orders of magnitude smaller than the initial 
clone.  Same goes for fetches to deepen a shallow clone which are in 
fact fetches going back in history instead of forward.  I still stands 
by my assertion that bundles are suboptimal for a restartable clone.

As for GitTorrent / git-mirror-sync... those are still vaporwares to me 
and I therefore have doubts about their actual feasability. So no, I 
don't count on them.

> > Hence having a restartable git-archive service to create the top 
> > revision with the ability to cheaply (in terms of network bandwidth) 
> > deepen the history afterwards is probably the most straight forward way 
> > to achieve that.  The server needs no be aware of separate bundles, etc.  
> > And the shared object store still works as usual with the same cached IO 
> > whether the data is needed for a traditional fetch or a "git archive" 
> > operation.
> 
> It's the "cheaply deepen history" that I doubt would be easy.  This is
> the most difficult part, I think (see also below).

Don't think so.  Try this:

	mkdir test
	cd test
	git init
	git fetch --depth=1 git://git.kernel.org/pub/scm/git/git.git

REsult:

remote: Counting objects: 1824, done.
remote: Compressing objects: 100% (1575/1575), done.
Receiving objects: 100% (1824/1824), 3.01 MiB | 975 KiB/s, done.
remote: Total 1824 (delta 299), reused 1165 (delta 180)
Resolving deltas: 100% (299/299), done.
From git://git.kernel.org/pub/scm/git/git
 * branch            HEAD       -> FETCH_HEAD

You'll get the very latest revision for HEAD, and only that.  The size 
of the transfer will be roughly the size of a daily snapshot, except it 
is fully up to date.  It is however non resumable in the event of a 
network outage.  My proposal is to replace this with a "git archive" 
call.  It won't get all branches, but for the purpose of initialising 
one's repository that should be good enough.  And the "git archive" can 
be fully resumable as I explained.

Now to deepen that history.  Let's say you want 10 more revisions going 
back then you simply perform the fetch again with a --depth=10.  Right 
now it doesn't seem to work optimally, but the pack that is then being 
sent could be made of deltas against objects found in the commits we 
already have.  Currently it seems that a pack that also includes those 
objects we already have in addition to those we want is created, which 
is IMHO a flaw in the shallow support that shouldn't be too hard to fix.  
Each level of deepening should then be as small as standard fetches 
going forward when updating the repository with new revisions.

> > Why "git archive"?  Because its content is well defined.  So if you give 
> > it a commit SHA1 you will always get the same stream of bytes (after 
> > decompression) since the way git sort files is strictly defined.  It is 
> > therefore easy to tell a remote "git archive" instance that we want the 
> > content for commit xyz but that we already got n files already, and that 
> > the last file we've got has m bytes.  There is simply no confusion about 
> > what we've got already, unlike with a partial pack which might need 
> > yet-to-be-received objects in order to make sense of what has been 
> > already received.  The server simply has to skip that many files and 
> > resume the transfer at that point, independently of the compression or 
> > even the archive format.
> 
> Let's reiterate it to check if I understand it correctly:
> 
> Any "restartable clone" / "resumable fetch" solution must begin with
> a file which is rock-solid stable wrt. reproductability given the same
> parameters.  git-archive has this feature, packfile doesn't (so I guess
> that bundle also doesn't, unless it was cached / saved on disk).

Right.

> It would be useful if it was possible to generate part of this rock-solid
> file for partial (range, resume) request, without need to generate 
> (calculate) parts that client already downloaded.  Otherwise server has
> to either waste disk space and IO for caching, or waste CPU (and IO)
> on generating part which is not needed and dropping it to /dev/null.
> git-archive you say has this feature.

"Could easily have" is more appropriate.

> Next you need to tell server that you have those objects got using
> resumable download part ("git archive HEAD" in your proposal), and
> that it can use them and do not include them in prepared file/pack.
> "have" is limited to commits, and "have <sha1>" tells server that
> you have <sha1> and all its prerequisites (dependences).  You can't 
> use "have <sha1>" with git-archive solution.  I don't know enough
> about 'shallow' capability (and what it enables) to know whether
> it can be used for that.  Can you elaborate?

See above, or Documentation/technical/shallow.txt.

> Then you have to finish clone / fetch.  All solutions so far include
> some kind of incremental improvements.  My first proposal of bisect
> fetching 1/nth or predefined size pack is buttom-up solution, where
> we build full clone from root commits up.  You propose, from what
> I understand build full clone from top commit down, using deepening
> from shallow clone.  In this step you either get full incremental
> or not; downloading incremental (from what I understand) is not
> resumable / they do not support partial fetch.

Right.  However, like I said, the incremental part should be much 
smaller and therefore less susceptible to network troubles.


Nicolas

^ permalink raw reply

* Re: question concerning branches
From: Jakub Narebski @ 2009-08-19 20:57 UTC (permalink / raw)
  To: Theodore Tso; +Cc: Ingo Brueckl, git
In-Reply-To: <20090819203917.GH27206@mit.edu>

Theodore Tso wrote:

> Personally, in the cases where I can't finish a commit before I need
> to switch away to another branch, my preference is to not use "git
> stash", but instead to create a topic branch, and then check in a
> partially completed change on the topic branch, which I can later
> ammend using "git commit --amend" (or if I have multiple commits on
> the topic branch, "git rebase --interactive").  This is because I can
> use the commit description to leave myself some notes about what still
> needs to be done before the commit can be finalized.

Errr... you are aware that you can use "git stash save <message>" (i.e. 
specify commit message for stash; well, the subject), don't you?

-- 
Jakub Narebski
Poland

^ permalink raw reply

* Re: question concerning branches
From: Theodore Tso @ 2009-08-19 20:39 UTC (permalink / raw)
  To: Ingo Brueckl; +Cc: Jakub Narebski, git
In-Reply-To: <4a8c51f5@wupperonline.de>

On Wed, Aug 19, 2009 at 09:45:00PM +0200, Ingo Brueckl wrote:
> Jakub Narebski <jnareb@gmail.com> writes:
> 
> > You finish old work (or stash it away), _then_ you begin new work.
> 
> Ok, this helps me a little bit to understand.
> 
> The branches aren't designed to split my work, but rather something to
> collect the different parts of my work.
> 
> But as software development often is something where you are coding on
> several issues at the same time which can't be committed immediately, it
> sounds that 'stash' is the developer's best friend.

Context switching has overhead; so it's usually better to try to
complete one task before switching to another.  Granted, sometimes it
can't be done, but it's something you should really try to do.

Also, commits are easier to review if they are kept small; if you
localize changes into separate commits, it's often easier to detet
problems when doing "git bisect", for example.  So if you are often
needing to switch while leaving something that isn't ready to be
committed, you might want to ask yourself if you are putting too many
changes into a single ocmmit.

Personally, in the cases where I can't finish a commit before I need
to switch away to another branch, my preference is to not use "git
stash", but instead to create a topic branch, and then check in a
partially completed change on the topic branch, which I can later
ammend using "git commit --amend" (or if I have multiple commits on
the topic branch, "git rebase --interactive").  This is because I can
use the commit description to leave myself some notes about what still
needs to be done before the commit can be finalized.

						- Ted

^ permalink raw reply

* Re: question concerning branches
From: Jakub Narebski @ 2009-08-19 20:01 UTC (permalink / raw)
  To: Ingo Brueckl; +Cc: git, Avery Pennarun
In-Reply-To: <4a8c51f5@wupperonline.de>

On Wed, 19 Aug 2009, Ingo Brueckl wrote:
> Jakub Narebski <jnareb@gmail.com> writes:
> 
> > You finish old work (or stash it away), _then_ you begin new work.
> 
> Ok, this helps me a little bit to understand.
> 
> The branches aren't designed to split my work, but rather something to
> collect the different parts of my work.

Well, git is flexible enough that it can support also the workflow you 
tried to use.  

Namely you can have many working directories tied to single repository 
(each of those checkouts should be of different branch).  You can use 
git-new-workdir script from contrib/worktree for that.  Then to switch 
branches you would just cd to appropriate directory (and keep unsaved 
changes and untracked files).  That said it is [much] less used 
workflow.
 
> But as software development often is something where you are coding on
> several issues at the same time which can't be committed immediately,
> it sounds that 'stash' is the developer's best friend.

Well, you can also commit and then clean up history with interactive 
rebase (or patch management interface such as StGit or Guilt).  In 
distributed version control systems like Git the act of publishing 
changes is separate from the act of committing them (you should not 
rewrite published history, though).

-- 
Jakub Narebski
Poland

^ permalink raw reply

* Re: question concerning branches
From: Jacob Helwig @ 2009-08-19 19:53 UTC (permalink / raw)
  To: Ingo Brueckl; +Cc: Jakub Narebski, git
In-Reply-To: <4a8c51f5@wupperonline.de>

On Wed, Aug 19, 2009 at 12:45, Ingo Brueckl<ib@wupperonline.de> wrote:
>
> But as software development often is something where you are coding on
> several issues at the same time which can't be committed immediately, it
> sounds that 'stash' is the developer's best friend.
>
> Ingo
>

There is no problem with having temporary commits on local branches,
however.

Quite frequently, I'll "git commit -a -m 'Temp commit'; git checkout
other-branch".  As long as you don't make these temporary commits
public, it's very easy to munge them (See: "git rebase --interactive
<commitish>", and "git reset --soft <commitish>").

-Jacob

^ permalink raw reply

* Re: question concerning branches
From: Avery Pennarun @ 2009-08-19 19:50 UTC (permalink / raw)
  To: Ingo Brueckl; +Cc: Jakub Narebski, git
In-Reply-To: <4a8c51f5@wupperonline.de>

On Wed, Aug 19, 2009 at 7:45 PM, Ingo Brueckl<ib@wupperonline.de> wrote:
> Jakub Narebski <jnareb@gmail.com> writes:
>
>> You finish old work (or stash it away), _then_ you begin new work.
>
> Ok, this helps me a little bit to understand.
>
> The branches aren't designed to split my work, but rather something to
> collect the different parts of my work.
>
> But as software development often is something where you are coding on
> several issues at the same time which can't be committed immediately, it
> sounds that 'stash' is the developer's best friend.

Or you could just 'commit' more frequently, but don't 'push' so you're
not disturbing anyone else until you're done.

This is a big difference from how centralized VCSs work: there, a
commit is a major operation that you're afraid to do in case you make
someone else mad.  In git, commits are cheap, you just need to be
careful about pushing.

(You can also clean up your series of commits before pushing by using
'git rebase')

Have fun,

Avery

^ permalink raw reply

* question concerning branches
From: Ingo Brueckl @ 2009-08-19 19:45 UTC (permalink / raw)
  To: Jakub Narebski; +Cc: git
In-Reply-To: <m33a7noc3u.fsf@localhost.localdomain>

Jakub Narebski <jnareb@gmail.com> writes:

> You finish old work (or stash it away), _then_ you begin new work.

Ok, this helps me a little bit to understand.

The branches aren't designed to split my work, but rather something to
collect the different parts of my work.

But as software development often is something where you are coding on
several issues at the same time which can't be committed immediately, it
sounds that 'stash' is the developer's best friend.

Ingo

^ permalink raw reply

* Re: Continue git clone after interruption
From: Jakub Narebski @ 2009-08-19 19:42 UTC (permalink / raw)
  To: Nicolas Pitre; +Cc: Tomasz Kontusz, git, Johannes Schindelin
In-Reply-To: <alpine.LFD.2.00.0908191326360.6044@xanadu.home>

Cc-ed Dscho, so he can easier participate in this subthread.

On Wed, 19 Aug 2009, Nicolas Pitre wrote:
> On Wed, 19 Aug 2009, Jakub Narebski wrote:

> > P.S. What do you think about 'bundle' capability extension mentioned
> >      in a side sub-thread?
> 
> I don't like it.  Reason is that it forces the server to be (somewhat) 
> stateful by having to keep track of those bundles and cycle them, and it 
> doubles the disk usage by having one copy of the repository in the form 
> of the original pack(s) and another copy as a bundle.

I agree about problems with disk usage, but I disagree about server
having to be stateful; server can just simply scan for bundles, and
offer links to them if client requests 'bundles' capability, somewhere
around initial git-ls-remote list of refs.

> Of course, the idea of having a cron job generating a bundle and 
> offering it for download through HTTP or the like is fine if people are 
> OK with that, and that requires zero modifications to git.  But I don't 
> think that is a solution that scales.

Well, offering daily bundle in addition to daily snapshot could be
a good practice, at least until git acquires resumable fetch (resumable
clone).

> 
> If you think about git.kernel.org which has maybe hundreds of 
> repositories where the big majority of them are actually forks of Linus' 
> own repository, then having all those forks reference Linus' repository 
> is a big disk space saver (and IO too as the referenced repository is 
> likely to remain cached in memory).  Having a bundle ready for each of 
> them will simply kill that space advantage, unless they all share the 
> same bundle.

I am thinking about sharing the same bundle for related projects.

> 
> Now sharing that common bundle could be done of course, but that makes 
> things yet more complex while still wasting IO because some requests 
> will hit the common pack and some others will hit the bundle, making 
> less efficient usage of the disk cache on the server.

Hmmm... true (unless bundles are on separate server).

> 
> Yet, that bundle would probably not contain the latest revision if it is 
> only periodically updated, even less so if it is shared between multiple 
> repositories as outlined above.  And what people with slow/unreliable 
> network links are probably most interested in is the latest revision and 
> maybe a few older revisions, but probably not the whole repository as 
> that is simply too long to wait for.  Hence having a big bundle is not 
> flexible either with regards to the actual data transfer size.

I agree that bundle would be useful for restartable clone, and not
useful for restartable fetch.  Well, unless you count (non-existing)
GitTorrent / git-mirror-sync as this solution... ;-)

> 
> Hence having a restartable git-archive service to create the top 
> revision with the ability to cheaply (in terms of network bandwidth) 
> deepen the history afterwards is probably the most straight forward way 
> to achieve that.  The server needs no be aware of separate bundles, etc.  
> And the shared object store still works as usual with the same cached IO 
> whether the data is needed for a traditional fetch or a "git archive" 
> operation.

It's the "cheaply deepen history" that I doubt would be easy.  This is
the most difficult part, I think (see also below).

> 
> Why "git archive"?  Because its content is well defined.  So if you give 
> it a commit SHA1 you will always get the same stream of bytes (after 
> decompression) since the way git sort files is strictly defined.  It is 
> therefore easy to tell a remote "git archive" instance that we want the 
> content for commit xyz but that we already got n files already, and that 
> the last file we've got has m bytes.  There is simply no confusion about 
> what we've got already, unlike with a partial pack which might need 
> yet-to-be-received objects in order to make sense of what has been 
> already received.  The server simply has to skip that many files and 
> resume the transfer at that point, independently of the compression or 
> even the archive format.

Let's reiterate it to check if I understand it correctly:


Any "restartable clone" / "resumable fetch" solution must begin with
a file which is rock-solid stable wrt. reproductability given the same
parameters.  git-archive has this feature, packfile doesn't (so I guess
that bundle also doesn't, unless it was cached / saved on disk).

It would be useful if it was possible to generate part of this rock-solid
file for partial (range, resume) request, without need to generate 
(calculate) parts that client already downloaded.  Otherwise server has
to either waste disk space and IO for caching, or waste CPU (and IO)
on generating part which is not needed and dropping it to /dev/null.
git-archive you say has this feature.

Next you need to tell server that you have those objects got using
resumable download part ("git archive HEAD" in your proposal), and
that it can use them and do not include them in prepared file/pack.
"have" is limited to commits, and "have <sha1>" tells server that
you have <sha1> and all its prerequisites (dependences).  You can't 
use "have <sha1>" with git-archive solution.  I don't know enough
about 'shallow' capability (and what it enables) to know whether
it can be used for that.  Can you elaborate?

Then you have to finish clone / fetch.  All solutions so far include
some kind of incremental improvements.  My first proposal of bisect
fetching 1/nth or predefined size pack is buttom-up solution, where
we build full clone from root commits up.  You propose, from what
I understand build full clone from top commit down, using deepening
from shallow clone.  In this step you either get full incremental
or not; downloading incremental (from what I understand) is not
resumable / they do not support partial fetch.

Do I understand this correctly?
-- 
Jakub Narebski
Poland

^ permalink raw reply

* question concerning branches
From: Ingo Brueckl @ 2009-08-19 19:21 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: git
In-Reply-To: <7vr5v7vehj.fsf@alter.siamese.dyndns.org>

Junio C Hamano <gitster@pobox.com> writes:

> This is one of the most useful features.

Wow. I'm sursprised to hear that, because I consider it at the moment as a
very strange one.

> For example, it is an essential
> part of supporting the workflow described here:
>     http://gitster.livejournal.com/25892.html

Here is what I'd expect to do with git (described with my own words, not in
git commands):

1. commit the quick fix to the release branch
2. push this single commit to origin and master

Now that all branches have the commit a later push and pull should notice
this and "skip" it.

This leads to a second question I have. Assuming I have three patches in my
repo (#1, #2 and #3), is it possible to push only #2 (because it is a
quick fix) and later, maybe after I committed #4, the rest, i.e. #1, #2 and
#4?

Ingo

^ permalink raw reply

* Re: question concerning branches
From: Jakub Narebski @ 2009-08-19 19:08 UTC (permalink / raw)
  To: Ingo Bruecki; +Cc: Avery Pennarun, git
In-Reply-To: <4a8c4425@wupperonline.de>

ib@wupperonline.de (Ingo Brueckl) writes:

> Avery Pennarun <apenwarr@gmail.com> writes:
> 
> > You seem to have forgotten the "git commit" step before switching back
> > to master.
> 
> No, I passed over the commit in my example. I know that after the commit the
> things are as they ought to be, but what if I can't do a commit because I am
> in the middle of coding and have to have a break?

Then you use git-stash.  It was invented for that.
 
> > You have a modified file in your repository; what did you *want* to happen
> > when you switched branches?
> 
> I want an unchanged file in master if I switch there (because I worked in a
> different branch) and a changed version in the test branch.
> 
> Why is the *master* different depending on whether my work in test in still
> going on or committed?!

Branches are about commits.  State of a working directory doesn't
belong to a branch (in Git).  Learning concepts behind Git would help
you in understanding it (Git is very consistent), which in turn would
help in using it.

What about untracked files?  Do you want to lose them when you switch
branches?

> 
> Actually, I cannot image how branches are practicable if I always have to
> have in mind possibly still uncommitted work. Shouldn't it be git's work
> to ensure that master will remain it was when branching?
> 
> Without git I'd make a copy for testing new features. With git, it seems that
> I have to do the same (a clone). This is what I don't understand.

You finish old work (or stash it away), _then_ you begin new work.

> 
> > (Many people find the current behaviour very convenient.)

Take the following example.  You started coding some feature on
'master' branch, then you realized that this feature is more
complicated than you thought at first, so it should be developed in
separate topic branch.  You do "git checkout -b featureA", and voila
you are now coding on feature branch 'featureA'.

> > You might also want to look at the "git stash" command.
> 
> Yes, but isn't it annoying to leave the test branch always either with stash
> or commit in order to have an unchanged master?!

No, it isn't.

-- 
Jakub Narebski

Git User's Survey 2009: http://tinyurl.com/GitSurvey2009

^ permalink raw reply

* Re: Continue git clone after interruption
From: Nicolas Pitre @ 2009-08-19 19:04 UTC (permalink / raw)
  To: Jakub Narebski; +Cc: Tomasz Kontusz, git
In-Reply-To: <200908191719.52974.jnareb@gmail.com>

On Wed, 19 Aug 2009, Jakub Narebski wrote:

> There are 114937 objects in this packfile, including 56249 objects
> used as base (can be deltified or not).  git-verify-pack -v shows
> that all objects have total size-in-packfile of 33 MB (which agrees
> with packfile size of 33 MB), with 17 MB size-in-packfile taken by
> deltaified objects, and 16 MB taken by base objects.
> 
>   git verify-pack -v | 
>     grep -v "^chain" | 
>     grep -v "objects/pack/pack-" > verify-pack.out
> 
>   sum=0; bsum=0; dsum=0; 
>   while read sha1 type size packsize off depth base; do
>     echo "$sha1" >> verify-pack.sha1.out
>     sum=$(( $sum + $packsize ))
>     if [ -n "$base" ]; then 
>        echo "$sha1" >> verify-pack.delta.out
>        dsum=$(( $dsum + $packsize ))
>     else
>        echo "$sha1" >> verify-pack.base.out
>        bsum=$(( $bsum + $packsize ))
>     fi
>   done < verify-pack.out
>   echo "sum=$sum; bsum=$bsum; dsum=$dsum"

Your object classification is misleading.  Because an object has no 
base, that doesn't mean it is necessarily a base itself.  You'd have to 
store $base into a separate file and then sort it and remove duplicates 
to know the actual number of base objects.  What you have right now is 
strictly delta objects and non-delta objects. And base objects can 
themselves be delta objects already of course.

Also... my git repo after 'git gc --aggressive' contains a pack which 
size is 22 MB.  Your script tells me:

sum=22930254; bsum=14142012; dsum=8788242

and:

   29558 verify-pack.base.out
   82043 verify-pack.delta.out
  111601 verify-pack.out
  111601 verify-pack.sha1.out

meaning that I have 111601 total objects, of which 29558 are non-deltas 
occupying 14 MB and 82043 are deltas occupying 8 MB.  That certainly 
shows how deltas are space efficient.  And with a minor modification to 
your script, I know that 44985 objects are actually used as a delta 
base.  So, on average, each base is responsible for nearly 2 deltas.

> >>>> (BTW what happens if this pack is larger than file size limit for 
> >>>> given filesystem?).
> [...]
> 
> >> If I remember correctly FAT28^W FAT32 has maximum file size of 2 GB.
> >> FAT is often used on SSD, on USB drive.  Although if you have  2 GB
> >> packfile, you are doing something wrong, or UGFWIINI (Using Git For
> >> What It Is Not Intended).
> > 
> > Hopefully you're not performing a 'git clone' off of a FAT filesystem.  
> > For physical transport you may repack with the appropriate switches.
> 
> Not off a FAT filesystem, but into a FAT filesystem.

That's what I meant, sorry.  My point still stands.

> > The front of the pack is the critical point.  If you get enough to 
> > create the top commit then further transfers can be done incrementally 
> > with only the deltas between each commits.
> 
> How?  You have some objects that can be used as base; how to tell 
> git-daemon that we have them (but not theirs prerequisites), and how
> to generate incrementals?

Just the same as when you perform a fetch to update your local copy of a 
remote branch: you tell the remote about the commit you have and the one 
you want, and git-repack will create delta objects for the commit you 
want against similar objects from the commit you already have, and skip 
those objects from the commit you want that are already included in the 
commit you have.

> >> A question about pack protocol negotiation.  If clients presents some
> >> objects as "have", server can and does assume that client has all 
> >> prerequisites for such objects, e.g. for tree objects that it has
> >> all objects for files and directories inside tree; for commit it means
> >> all ancestors and all objects in snapshot (have top tree, and its 
> >> prerequisites).  Do I understand this correctly?
> > 
> > That works only for commits.
> 
> Hmmmm... how do you intent for "prefetch top objects restartable-y first"
> to work, then?

See my latest reply to dscho (you were in CC already).

> >> BTW. because of compression it might be more difficult to resume 
> >> archive creation in the middle, I think...
> > 
> > Why so?  the tar+gzip format is streamable.
> 
> gzip format uses sliding window in compression.  "cat a b | gzip"
> is different from "cat <(gzip a) <(gzip b)".
> 
> But that doesn't matter.  If we are interrupted in the middle, we can
> uncompress what we have to check how far did we get, and tell server
> to send the rest; this way server wouldn't have to even generate 
> (but not send) what we get as partial transfer.

You got it.

> P.S. What do you think about 'bundle' capability extension mentioned
>      in a side sub-thread?

I don't like it.  Reason is that it forces the server to be (somewhat) 
stateful by having to keep track of those bundles and cycle them, and it 
doubles the disk usage by having one copy of the repository in the form 
of the original pack(s) and another copy as a bundle.

Of course, the idea of having a cron job generating a bundle and 
offering it for download through HTTP or the like is fine if people are 
OK with that, and that requires zero modifications to git.  But I don't 
think that is a solution that scales.

If you think about git.kernel.org which has maybe hundreds of 
repositories where the big majority of them are actually forks of Linus' 
own repository, then having all those forks reference Linus' repository 
is a big disk space saver (and IO too as the referenced repository is 
likely to remain cached in memory).  Having a bundle ready for each of 
them will simply kill that space advantage, unless they all share the 
same bundle.

Now sharing that common bundle could be done of course, but that makes 
things yet more complex while still wasting IO because some requests 
will hit the common pack and some others will hit the bundle, making 
less efficient usage of the disk cache on the server.

Yet, that bundle would probably not contain the latest revision if it is 
only periodically updated, even less so if it is shared between multiple 
repositories as outlined above.  And what people with slow/unreliable 
network links are probably most interested in is the latest revision and 
maybe a few older revisions, but probably not the whole repository as 
that is simply too long to wait for.  Hence having a big bundle is not 
flexible either with regards to the actual data transfer size.

Hence having a restartable git-archive service to create the top 
revision with the ability to cheaply (in terms of network bandwidth) 
deepen the history afterwards is probably the most straight forward way 
to achieve that.  The server needs no be aware of separate bundles, etc.  
And the shared object store still works as usual with the same cached IO 
whether the data is needed for a traditional fetch or a "git archive" 
operation.

Why "git archive"?  Because its content is well defined.  So if you give 
it a commit SHA1 you will always get the same stream of bytes (after 
decompression) since the way git sort files is strictly defined.  It is 
therefore easy to tell a remote "git archive" instance that we want the 
content for commit xyz but that we already got n files already, and that 
the last file we've got has m bytes.  There is simply no confusion about 
what we've got already, unlike with a partial pack which might need 
yet-to-be-received objects in order to make sense of what has been 
already received.  The server simply has to skip that many files and 
resume the transfer at that point, independently of the compression or 
even the archive format.


Nicolas

^ permalink raw reply


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox