* uncommitted changes in a renamed directory are clobbered on merge
From: Paul Grayson @ 2011-11-10 5:28 UTC (permalink / raw)
To: git
Hello,
While merging with uncommitted changes is strongly discouraged, Git
generally tries to not overwrite your work without warning. However,
there are some situations where Git silently erases uncommitted
changes. Specifically, it seems that if you rename a directory on one
branch, make uncommitted changes to files within that branch, then
merge in another branch, files that would conflict get replaced by the
version on the other branch. We have observed this on Git versions
1.7.5.4 and 1.7.7.3.
I have placed a script demonstrating this problem at
https://gist.github.com/1354160
Here is a shell transcript showing what it looks like:
gitbug$ git init single
Initialized empty Git repository in /home/paul/gitprojects/gitbug/single/.git/
gitbug$ cd single/
gitbug/single$ mkdir test
gitbug/single$ echo "hi" > test/test.txt
gitbug/single$ git add .
gitbug/single$ git commit -am "initial revision"
[master (root-commit) 8be9a1e] initial revision
1 files changed, 1 insertions(+), 0 deletions(-)
create mode 100644 test/test.txt
gitbug/single$ git checkout -b branch1
Switched to a new branch 'branch1'
gitbug/single$ git mv test/ test2
gitbug/single$ git commit -am "renamed test to test2"
[branch1 80d497c] renamed test to test2
1 files changed, 0 insertions(+), 0 deletions(-)
rename {test => test2}/test.txt (100%)
gitbug/single$ git checkout master
Switched to branch 'master'
gitbug/single$ echo "change on master" >> test/test.txt
gitbug/single$ git commit -am "made a change on master"
[master 51d3a75] made a change on master
1 files changed, 1 insertions(+), 0 deletions(-)
gitbug/single$ git checkout branch1
Switched to branch 'branch1'
gitbug/single$ echo "change on branch1" >> test2/test.txt
gitbug/single$ git merge master
Auto-merging test2/test.txt
Merge made by recursive.
test2/test.txt | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
gitbug/single$ cat test2/test.txt
hi
change on master
gitbug/single$
I expected the merge to fail as it usually does when there is a
conflict with uncommitted changes. Instead, it silently deleted the
changes in test.txt.
One of our developers just lost a few hours of work to this bug. Please fix!
Sincerely,
Paul Grayson
^ permalink raw reply
* Re: git-svn: t9155 fails against subversion 1.7.0
From: Frans Klaver @ 2011-11-10 6:02 UTC (permalink / raw)
To: git@vger.kernel.org
In-Reply-To: <op.v4neh4q20aolir@keputer>
I missed $gmane/184644 in my search for this issue.
On Tue, 08 Nov 2011 23:09:30 +0100, Frans Klaver <fransklaver@gmail.com>
wrote:
> For kicks I decided to run the tests and noticed that on master
> t9155-git-svn-fetch-deleted-tag fails against svn 1.7.0. We hit an
> assertion in subversion's dirent_uri.c, stating that we don't provide a
> canonical url. I haven't tested against other subversion versions. I
> dare assume that this issue doesn't arise on earlier versions. It
> probably won't affect a lot of users right now, but it will in the
> future.
>
> Here's some verbose test output:
> expecting success:
> git svn init --stdlayout "$svnrepo" git_project &&
> cd git_project &&
> git svn fetch &&
>
> git diff --exit-code mybranch:trunk/subdir/file tags/mytag:file &&
> git diff --exit-code master:subdir/file tags/mytag^:file
>
> Initialized empty Git repository in /home/frans/devsw/git/t/trash
> directory.t9155-git-svn-fetch-deleted-tag/git_project/.git/
> svn: E235000: In file 'subversion/libsvn_subr/dirent_uri.c' line 2291:
> assertion failed (svn_uri_is_canonical(url, pool))
> error: git-svn died of signal 6
> not ok - 2 fetch deleted tags from same revision with checksum error
>
> I've been trying to debug and got down to:
> Git::SVN::Ra::new(/home/frans/devsw/git/git-svn:5496):
> 5496: my $self = SVN::Ra->new(url => escape_url($url), auth => $baton,
> 5497: config => $config,
> 5498: pool => SVN::Pool->new,
> 5499: auth_provider_callbacks => $callbacks);
> ...
> SVN::Ra::new(/usr/lib/perl5/vendor_perl/5.12.4/i686-linux/SVN/Ra.pm:529):
> 529: $self->{session} = SVN::_Ra::svn_ra_open($self->{url},
> $callback, $self->{config} || {}, $pool);
> DB<3> p $self->{url}
> file:///home/frans/devsw/git/t/trash
> directory.t9155-git-svn-fetch-deleted-tag/svnrepo
>
> The url looks like that throughout the stack (as far as I've seen), so
> if it is wrong, it is probably wrong at top-level. Hope someone with a
> bit more experience knows how to deal with this.
>
> Thanks,
> Frans
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
^ permalink raw reply
* Re: Bash tab completion for _git_fetch alias is broken on Git 1.7.7.1
From: Johannes Sixt @ 2011-11-10 7:09 UTC (permalink / raw)
To: nathan.f77; +Cc: Junio C Hamano, git
In-Reply-To: <7vehxgu0fy.fsf@alter.siamese.dyndns.org>
Am 11/10/2011 4:21, schrieb Junio C Hamano:
> Nathan Broadbent <nathan.f77@gmail.com> writes:
>
>> Dear git mailing list,
>>
>> I'm assigning the `_git_fetch` bash tab completion to the alias `gf`,
>> with the following command:
>>
>> complete -o default -o nospace -F _git_fetch gf
>>
>> The tab completion then works fine in git 1.7.0.4, but breaks on git
>> 1.7.7.1, with the following error:
>
> We have been cooking for 1.7.8 and have the first release candidate
> 1.7.8-rc1; could you try it and report what you find out?
It looks like _git_fetch is not meant to be called directly. All git
completions must go through _git.
See also this post:
http://thread.gmane.org/gmane.comp.version-control.msysgit/13310/focus=13335
-- Hannes
^ permalink raw reply
* [PATCH 1/6] Print trace in rev-list machinery
From: Nguyễn Thái Ngọc Duy @ 2011-11-10 7:12 UTC (permalink / raw)
To: git; +Cc: Nguyễn Thái Ngọc Duy
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
revision.c | 5 +++++
1 files changed, 5 insertions(+), 0 deletions(-)
diff --git a/revision.c b/revision.c
index 8764dde..0aa3638 100644
--- a/revision.c
+++ b/revision.c
@@ -1690,6 +1690,11 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, struct s
if (opt)
submodule = opt->submodule;
+ if (argc)
+ trace_argv_printf(argv + 1, "trace: rev-list");
+ else
+ trace_printf("trace: rev-list\n");
+
/* First, search for "--" */
seen_dashdash = 0;
for (i = 1; i < argc; i++) {
--
1.7.4.74.g639db
^ permalink raw reply related
* [PATCH 2/6] find_pack_entry(): do not keep packed_git pointer locally
From: Nguyễn Thái Ngọc Duy @ 2011-11-10 7:12 UTC (permalink / raw)
To: git; +Cc: Nguyễn Thái Ngọc Duy
In-Reply-To: <1320909155-4575-1-git-send-email-pclouds@gmail.com>
Commit f7c22cc (always start looking up objects in the last used pack
first - 2007-05-30) introduce a static packed_git* pointer as an
optimization.
However keeping the pointer may become invalid if free_pack_by_name()
happens to free that particular pack.
Move the pointer out and reset the pointer in free_pack_by_name() if we
free it.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
sha1_file.c | 13 ++++++++-----
1 files changed, 8 insertions(+), 5 deletions(-)
diff --git a/sha1_file.c b/sha1_file.c
index 3401301..155c808 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -53,6 +53,8 @@ static struct cached_object empty_tree = {
0
};
+static struct packed_git *find_pack_entry_last_found = (void *)1;
+
static struct cached_object *find_cached_object(const unsigned char *sha1)
{
int i;
@@ -719,6 +721,8 @@ void free_pack_by_name(const char *pack_name)
close_pack_index(p);
free(p->bad_object_sha1);
*pp = p->next;
+ if (find_pack_entry_last_found == p)
+ find_pack_entry_last_found = (void*)1;
free(p);
return;
}
@@ -2010,14 +2014,13 @@ static int is_pack_valid(struct packed_git *p)
static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
{
- static struct packed_git *last_found = (void *)1;
struct packed_git *p;
off_t offset;
prepare_packed_git();
if (!packed_git)
return 0;
- p = (last_found == (void *)1) ? packed_git : last_found;
+ p = (find_pack_entry_last_found == (void *)1) ? packed_git : find_pack_entry_last_found;
do {
if (p->num_bad_objects) {
@@ -2044,16 +2047,16 @@ static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
e->offset = offset;
e->p = p;
hashcpy(e->sha1, sha1);
- last_found = p;
+ find_pack_entry_last_found = p;
return 1;
}
next:
- if (p == last_found)
+ if (p == find_pack_entry_last_found)
p = packed_git;
else
p = p->next;
- if (p == last_found)
+ if (p == find_pack_entry_last_found)
p = p->next;
} while (p);
return 0;
--
1.7.4.74.g639db
^ permalink raw reply related
* [PATCH 3/6] parse-options: add OPT_ULONG
From: Nguyễn Thái Ngọc Duy @ 2011-11-10 7:12 UTC (permalink / raw)
To: git; +Cc: Nguyễn Thái Ngọc Duy
In-Reply-To: <1320909155-4575-1-git-send-email-pclouds@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
parse-options.c | 15 +++++++++++++++
parse-options.h | 2 ++
2 files changed, 17 insertions(+), 0 deletions(-)
diff --git a/parse-options.c b/parse-options.c
index f0098eb..58bb83d 100644
--- a/parse-options.c
+++ b/parse-options.c
@@ -144,6 +144,21 @@ static int get_value(struct parse_opt_ctx_t *p,
return opterror(opt, "expects a numerical value", flags);
return 0;
+ case OPTION_ULONG:
+ if (unset) {
+ *(unsigned long *)opt->value = 0;
+ return 0;
+ }
+ if (opt->flags & PARSE_OPT_OPTARG && !p->opt) {
+ *(unsigned long *)opt->value = opt->defval;
+ return 0;
+ }
+ if (get_arg(p, opt, flags, &arg))
+ return -1;
+ if (!git_parse_ulong(arg, (unsigned long *)opt->value))
+ return opterror(opt, "expects a numerical value", flags);
+ return 0;
+
default:
die("should not happen, someone must be hit on the forehead");
}
diff --git a/parse-options.h b/parse-options.h
index 2e811dc..12294d7 100644
--- a/parse-options.h
+++ b/parse-options.h
@@ -16,6 +16,7 @@ enum parse_opt_type {
/* options with arguments (usually) */
OPTION_STRING,
OPTION_INTEGER,
+ OPTION_ULONG,
OPTION_CALLBACK,
OPTION_LOWLEVEL_CALLBACK,
OPTION_FILENAME
@@ -133,6 +134,7 @@ struct option {
#define OPT_SET_PTR(s, l, v, h, p) { OPTION_SET_PTR, (s), (l), (v), NULL, \
(h), PARSE_OPT_NOARG, NULL, (p) }
#define OPT_INTEGER(s, l, v, h) { OPTION_INTEGER, (s), (l), (v), "n", (h) }
+#define OPT_ULONG(s, l, v, h) { OPTION_ULONG, (s), (l), (v), "n", (h) }
#define OPT_STRING(s, l, v, a, h) { OPTION_STRING, (s), (l), (v), (a), (h) }
#define OPT_STRING_LIST(s, l, v, a, h) \
{ OPTION_CALLBACK, (s), (l), (v), (a), \
--
1.7.4.74.g639db
^ permalink raw reply related
* [PATCH 4/6] pack-objects: use parse_options()
From: Nguyễn Thái Ngọc Duy @ 2011-11-10 7:12 UTC (permalink / raw)
To: git; +Cc: Nguyễn Thái Ngọc Duy
In-Reply-To: <1320909155-4575-1-git-send-email-pclouds@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
There is a side effect of this conversion: --window and --depth now
can take units because git_parse_ulong() is always used by
OPTION_ULONG
builtin/pack-objects.c | 276 +++++++++++++++++++-----------------------------
1 files changed, 110 insertions(+), 166 deletions(-)
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 2b18de5..c1ca748 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -18,16 +18,11 @@
#include "refs.h"
#include "thread-utils.h"
-static const char pack_usage[] =
- "git pack-objects [ -q | --progress | --all-progress ]\n"
- " [--all-progress-implied]\n"
- " [--max-pack-size=<n>] [--local] [--incremental]\n"
- " [--window=<n>] [--window-memory=<n>] [--depth=<n>]\n"
- " [--no-reuse-delta] [--no-reuse-object] [--delta-base-offset]\n"
- " [--threads=<n>] [--non-empty] [--revs [--unpacked | --all]]\n"
- " [--reflog] [--stdout | base-name] [--include-tag]\n"
- " [--keep-unreachable | --unpack-unreachable]\n"
- " [< ref-list | < object-list]";
+static const char *pack_usage[] = {
+ "git pack-objects --stdout [options...] [< ref-list | < object-list]",
+ "git pack-objects [options...] base-name [< ref-list | < object-list]",
+ NULL
+};
struct object_entry {
struct pack_idx_entry idx;
@@ -2255,15 +2250,86 @@ static void get_object_list(int ac, const char **av)
loosen_unused_packed_objects(&revs);
}
+static int option_parse_index_version(const struct option *opt,
+ const char *arg, int unset)
+{
+ char *c;
+ const char *val = arg;
+ pack_idx_opts.version = strtoul(val, &c, 10);
+ if (pack_idx_opts.version > 2)
+ die("bad %s", val);
+ if (*c == ',')
+ pack_idx_opts.off32_limit = strtoul(c+1, &c, 0);
+ if (*c || pack_idx_opts.off32_limit & 0x80000000)
+ die("bad %s", val);
+ return 0;
+}
+
int cmd_pack_objects(int argc, const char **argv, const char *prefix)
{
int use_internal_rev_list = 0;
int thin = 0;
int all_progress_implied = 0;
- uint32_t i;
const char **rp_av;
int rp_ac_alloc = 64;
int rp_ac;
+ struct option pack_objects_options[] = {
+ OPT_SET_INT('q', NULL, &progress,
+ "do not show progress meter", 0),
+ OPT_SET_INT(0, "progress", &progress,
+ "show progress meter", 1),
+ OPT_SET_INT(0, "all-progress", &progress,
+ "show progress meter during object writing phase", 2),
+ OPT_BOOL(0, "all-progress-implied",
+ &all_progress_implied,
+ "similar to --all-progress when progress meter is shown"),
+ { OPTION_CALLBACK, 0, "index-version", NULL, "version",
+ "force generating pack index at a particular version",
+ 0, option_parse_index_version },
+ OPT_ULONG(0, "max-pack-size", &pack_size_limit,
+ "maximum size of each output pack file"),
+ OPT_BOOL(0, "local", &local,
+ "ignore borrowed objects from alternate object store"),
+ OPT_BOOL(0, "incremental", &incremental,
+ "ignore packed objects"),
+ OPT_ULONG(0, "window", &window, "limit pack window by objects"),
+ OPT_ULONG(0, "window-memory", &window_memory_limit,
+ "limit pack window by memory"),
+ OPT_INTEGER(0, "depth", &depth,
+ "limit pack window by maximum delta depth"),
+ OPT_BOOL(0, "reuse-delta", &reuse_delta,
+ "reusing existing deltas"),
+ OPT_BOOL(0, "reuse-object", &reuse_object,
+ "reusing existing objects"),
+ OPT_BOOL(0, "delta-base-offset", &allow_ofs_delta,
+ "use OFS_DELTA objects"),
+ OPT_INTEGER(0, "threads", &delta_search_threads,
+ "use threads when searching for best delta matches"),
+ OPT_BOOL(0, "non-empty", &non_empty,
+ "only create if it would contain at least one object"),
+ OPT_BOOL(0, "revs", &use_internal_rev_list,
+ "read revision arguments from standard output"),
+ OPT_ARGUMENT("unpacked", "limit the objects to those that are not already packed"),
+ OPT_ARGUMENT("all", "include all refs under $GIT_DIR/refs directory"),
+ OPT_ARGUMENT("reflog", "include objects referred by reflog entries"),
+ OPT_BOOL(0, "stdout", &pack_to_stdout,
+ "output pack to stdout"),
+ OPT_BOOL(0, "include-tag", &include_tag,
+ "include unasked-for annotated tags"),
+ OPT_BOOL(0, "keep-unreachable", &keep_unreachable,
+ "keep unreachable objects"),
+ OPT_BOOL(0, "unpack-unreachable", &unpack_unreachable,
+ "unpack unreachable objects"),
+ OPT_BOOL(0, "thin", &thin,
+ "create thin packs"),
+ OPT_BOOL(0, "honor-pack-keep", &ignore_packed_keep,
+ "ignore packs that have companion .keep file"),
+ OPT_INTEGER(0, "compression", &pack_compression_level,
+ "pack compression level"),
+ OPT_SET_INT(0, "keep-true-parents", &grafts_replace_parents,
+ "do not hide commits by grafts", 0),
+ OPT_END(),
+ };
read_replace_refs = 0;
@@ -2279,160 +2345,35 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
pack_compression_level = core_compression_level;
progress = isatty(2);
- for (i = 1; i < argc; i++) {
- const char *arg = argv[i];
-
- if (*arg != '-')
- break;
+ argc = parse_options(argc, argv, prefix, pack_objects_options,
+ pack_usage, 0);
- if (!strcmp("--non-empty", arg)) {
- non_empty = 1;
- continue;
- }
- if (!strcmp("--local", arg)) {
- local = 1;
- continue;
- }
- if (!strcmp("--incremental", arg)) {
- incremental = 1;
- continue;
- }
- if (!strcmp("--honor-pack-keep", arg)) {
- ignore_packed_keep = 1;
- continue;
- }
- if (!prefixcmp(arg, "--compression=")) {
- char *end;
- int level = strtoul(arg+14, &end, 0);
- if (!arg[14] || *end)
- usage(pack_usage);
- if (level == -1)
- level = Z_DEFAULT_COMPRESSION;
- else if (level < 0 || level > Z_BEST_COMPRESSION)
- die("bad pack compression level %d", level);
- pack_compression_level = level;
- continue;
+ while (argc && argv[0][0] == '-') {
+ use_internal_rev_list = 1;
+ if (rp_ac >= rp_ac_alloc - 1) {
+ rp_ac_alloc = alloc_nr(rp_ac_alloc);
+ rp_av = xrealloc(rp_av,
+ rp_ac_alloc * sizeof(*rp_av));
}
- if (!prefixcmp(arg, "--max-pack-size=")) {
- pack_size_limit_cfg = 0;
- if (!git_parse_ulong(arg+16, &pack_size_limit))
- usage(pack_usage);
- continue;
- }
- if (!prefixcmp(arg, "--window=")) {
- char *end;
- window = strtoul(arg+9, &end, 0);
- if (!arg[9] || *end)
- usage(pack_usage);
- continue;
- }
- if (!prefixcmp(arg, "--window-memory=")) {
- if (!git_parse_ulong(arg+16, &window_memory_limit))
- usage(pack_usage);
- continue;
- }
- if (!prefixcmp(arg, "--threads=")) {
- char *end;
- delta_search_threads = strtoul(arg+10, &end, 0);
- if (!arg[10] || *end || delta_search_threads < 0)
- usage(pack_usage);
+ rp_av[rp_ac++] = *argv;
+ argv++;
+ argc--;
+ }
+
+ if (!reuse_object)
+ reuse_delta = 0;
+ if (thin) {
+ use_internal_rev_list = 1;
+ rp_av[1] = "--objects-edge";
+ }
+ if (pack_compression_level == -1)
+ pack_compression_level = Z_DEFAULT_COMPRESSION;
+ else if (pack_compression_level < 0 || pack_compression_level > Z_BEST_COMPRESSION)
+ die("bad pack compression level %d", pack_compression_level);
#ifdef NO_PTHREADS
- if (delta_search_threads != 1)
- warning("no threads support, "
- "ignoring %s", arg);
+ if (delta_search_threads != 1)
+ warning("no threads support, ignoring %s", arg);
#endif
- continue;
- }
- if (!prefixcmp(arg, "--depth=")) {
- char *end;
- depth = strtoul(arg+8, &end, 0);
- if (!arg[8] || *end)
- usage(pack_usage);
- continue;
- }
- if (!strcmp("--progress", arg)) {
- progress = 1;
- continue;
- }
- if (!strcmp("--all-progress", arg)) {
- progress = 2;
- continue;
- }
- if (!strcmp("--all-progress-implied", arg)) {
- all_progress_implied = 1;
- continue;
- }
- if (!strcmp("-q", arg)) {
- progress = 0;
- continue;
- }
- if (!strcmp("--no-reuse-delta", arg)) {
- reuse_delta = 0;
- continue;
- }
- if (!strcmp("--no-reuse-object", arg)) {
- reuse_object = reuse_delta = 0;
- continue;
- }
- if (!strcmp("--delta-base-offset", arg)) {
- allow_ofs_delta = 1;
- continue;
- }
- if (!strcmp("--stdout", arg)) {
- pack_to_stdout = 1;
- continue;
- }
- if (!strcmp("--revs", arg)) {
- use_internal_rev_list = 1;
- continue;
- }
- if (!strcmp("--keep-unreachable", arg)) {
- keep_unreachable = 1;
- continue;
- }
- if (!strcmp("--unpack-unreachable", arg)) {
- unpack_unreachable = 1;
- continue;
- }
- if (!strcmp("--include-tag", arg)) {
- include_tag = 1;
- continue;
- }
- if (!strcmp("--unpacked", arg) ||
- !strcmp("--reflog", arg) ||
- !strcmp("--all", arg)) {
- use_internal_rev_list = 1;
- if (rp_ac >= rp_ac_alloc - 1) {
- rp_ac_alloc = alloc_nr(rp_ac_alloc);
- rp_av = xrealloc(rp_av,
- rp_ac_alloc * sizeof(*rp_av));
- }
- rp_av[rp_ac++] = arg;
- continue;
- }
- if (!strcmp("--thin", arg)) {
- use_internal_rev_list = 1;
- thin = 1;
- rp_av[1] = "--objects-edge";
- continue;
- }
- if (!prefixcmp(arg, "--index-version=")) {
- char *c;
- pack_idx_opts.version = strtoul(arg + 16, &c, 10);
- if (pack_idx_opts.version > 2)
- die("bad %s", arg);
- if (*c == ',')
- pack_idx_opts.off32_limit = strtoul(c+1, &c, 0);
- if (*c || pack_idx_opts.off32_limit & 0x80000000)
- die("bad %s", arg);
- continue;
- }
- if (!strcmp(arg, "--keep-true-parents")) {
- grafts_replace_parents = 0;
- continue;
- }
- usage(pack_usage);
- }
/* Traditionally "pack-objects [options] base extra" failed;
* we would however want to take refs parameter that would
@@ -2447,11 +2388,14 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
* walker.
*/
- if (!pack_to_stdout)
- base_name = argv[i++];
-
- if (pack_to_stdout != !base_name)
- usage(pack_usage);
+ if (!pack_to_stdout) {
+ if (!argc)
+ die("base name required if --stdout is not given");
+ base_name = argv[0];
+ argc--;
+ }
+ if (argc)
+ die("base name or --stdout are mutually exclusive");
if (!pack_to_stdout && !pack_size_limit)
pack_size_limit = pack_size_limit_cfg;
--
1.7.4.74.g639db
^ permalink raw reply related
* [PATCH 5/6] parse-options: allow OPTION_ARGUMENT to take argument
From: Nguyễn Thái Ngọc Duy @ 2011-11-10 7:12 UTC (permalink / raw)
To: git; +Cc: Nguyễn Thái Ngọc Duy
In-Reply-To: <1320909155-4575-1-git-send-email-pclouds@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
parse-options.c | 18 +++++++++++++++---
1 files changed, 15 insertions(+), 3 deletions(-)
diff --git a/parse-options.c b/parse-options.c
index 58bb83d..3a1575a 100644
--- a/parse-options.c
+++ b/parse-options.c
@@ -218,11 +218,23 @@ static int parse_long_opt(struct parse_opt_ctx_t *p, const char *arg,
if (options->type == OPTION_ARGUMENT) {
if (!rest)
continue;
- if (*rest == '=')
- return opterror(options, "takes no value", flags);
+ if (*rest == '=') {
+ if (options->flags & PARSE_OPT_NOARG)
+ return opterror(options, "takes no value", flags);
+ p->out[p->cpidx++] = *p->argv;
+ return 0;
+ }
+
if (*rest)
continue;
- p->out[p->cpidx++] = arg - 2;
+ p->out[p->cpidx++] = *p->argv;
+ if (!(options->flags & PARSE_OPT_NOARG)) {
+ p->argv++;
+ p->argc--;
+ if (!*p->argv)
+ return opterror(options, "takes no value", flags);
+ p->out[p->cpidx++] = *p->argv;
+ }
return 0;
}
if (!rest) {
--
1.7.4.74.g639db
^ permalink raw reply related
* [PATCH 6/6] Build in git-repack
From: Nguyễn Thái Ngọc Duy @ 2011-11-10 7:12 UTC (permalink / raw)
To: git; +Cc: Nguyễn Thái Ngọc Duy
In-Reply-To: <1320909155-4575-1-git-send-email-pclouds@gmail.com>
pack-objects learns a few more options to take over what's been done
by git-repack.sh. cmd_repack() becomes a wrapper around
cmd_pack_objects().
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
Makefile | 2 +-
builtin.h | 1 +
builtin/pack-objects.c | 297 +++++++++++++++++++++++++++++++++++++++-
contrib/examples/git-repack.sh | 186 +++++++++++++++++++++++++
git-repack.sh | 186 -------------------------
git.c | 1 +
6 files changed, 483 insertions(+), 190 deletions(-)
create mode 100755 contrib/examples/git-repack.sh
delete mode 100755 git-repack.sh
diff --git a/Makefile b/Makefile
index 17404c4..164052d 100644
--- a/Makefile
+++ b/Makefile
@@ -374,7 +374,6 @@ SCRIPT_SH += git-mergetool.sh
SCRIPT_SH += git-pull.sh
SCRIPT_SH += git-quiltimport.sh
SCRIPT_SH += git-rebase.sh
-SCRIPT_SH += git-repack.sh
SCRIPT_SH += git-request-pull.sh
SCRIPT_SH += git-stash.sh
SCRIPT_SH += git-submodule.sh
@@ -460,6 +459,7 @@ BUILT_INS += git-init$X
BUILT_INS += git-merge-subtree$X
BUILT_INS += git-peek-remote$X
BUILT_INS += git-repo-config$X
+BUILT_INS += git-repack$X
BUILT_INS += git-show$X
BUILT_INS += git-stage$X
BUILT_INS += git-status$X
diff --git a/builtin.h b/builtin.h
index 0a5b511..2a890ce 100644
--- a/builtin.h
+++ b/builtin.h
@@ -115,6 +115,7 @@ extern int cmd_reflog(int argc, const char **argv, const char *prefix);
extern int cmd_remote(int argc, const char **argv, const char *prefix);
extern int cmd_remote_ext(int argc, const char **argv, const char *prefix);
extern int cmd_remote_fd(int argc, const char **argv, const char *prefix);
+extern int cmd_repack(int argc, const char **argv, const char *prefix);
extern int cmd_repo_config(int argc, const char **argv, const char *prefix);
extern int cmd_rerere(int argc, const char **argv, const char *prefix);
extern int cmd_reset(int argc, const char **argv, const char *prefix);
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index c1ca748..7acec4c 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -17,10 +17,17 @@
#include "progress.h"
#include "refs.h"
#include "thread-utils.h"
+#include "sigchain.h"
static const char *pack_usage[] = {
"git pack-objects --stdout [options...] [< ref-list | < object-list]",
"git pack-objects [options...] base-name [< ref-list | < object-list]",
+ "git pack-objects --repack [options...]",
+ NULL
+};
+
+static char const * const repack_usage[] = {
+ "git repack [options]",
NULL
};
@@ -101,6 +108,15 @@ static struct object_entry *locate_object_entry(const unsigned char *sha1);
static uint32_t written, written_delta;
static uint32_t reused, reused_delta;
+#define REPACK_IN_PROGRESS (1 << 0)
+#define REPACK_UPDATE_INFO (1 << 1)
+#define REPACK_ALL_INTO_ONE (1 << 2)
+#define REPACK_REMOVE_REDUNDANT (1 << 3)
+
+static int repack_flags, nr_written_packs;
+static int repack_usedeltabaseoffset;
+static struct string_list written_packs;
+static struct string_list backup_files;
static void *get_delta(struct object_entry *entry)
{
@@ -561,6 +577,19 @@ static struct object_entry **compute_write_order(void)
return wo;
}
+static void backup_file(const char *path)
+{
+ struct stat st;
+ if (repack_flags & REPACK_IN_PROGRESS &&
+ !stat(path, &st)) {
+ struct strbuf old = STRBUF_INIT;
+ strbuf_addf(&old, "%s.old", path);
+ if (rename(path, old.buf))
+ die_errno("unable to rename pack %s", path);
+ string_list_append(&backup_files, strbuf_detach(&old, NULL));
+ }
+}
+
static void write_pack_file(void)
{
uint32_t i = 0, j;
@@ -632,6 +661,7 @@ static void write_pack_file(void)
free_pack_by_name(tmpname);
if (adjust_shared_perm(pack_tmp_name))
die_errno("unable to make temporary pack file readable");
+ backup_file(tmpname);
if (rename(pack_tmp_name, tmpname))
die_errno("unable to rename temporary pack file");
@@ -660,12 +690,22 @@ static void write_pack_file(void)
base_name, sha1_to_hex(sha1));
if (adjust_shared_perm(idx_tmp_name))
die_errno("unable to make temporary index file readable");
+ backup_file(tmpname);
if (rename(idx_tmp_name, tmpname))
die_errno("unable to rename temporary index file");
free((void *) idx_tmp_name);
free(pack_tmp_name);
- puts(sha1_to_hex(sha1));
+ if (repack_flags & REPACK_IN_PROGRESS) {
+ int len = strlen(tmpname);
+ char *s = xmalloc(len + 2);
+ memcpy(s, tmpname, len - 4);
+ memcpy(s + len - 4, ".pack", 6);
+ string_list_append(&written_packs, s);
+ nr_written_packs++;
+ }
+ else
+ puts(sha1_to_hex(sha1));
}
/* mark written objects as written to previous pack */
@@ -2222,7 +2262,8 @@ static void get_object_list(int ac, const char **av)
save_commit_buffer = 0;
setup_revisions(ac, av, &revs, NULL);
- while (fgets(line, sizeof(line), stdin) != NULL) {
+ while (!(repack_flags & REPACK_IN_PROGRESS) &&
+ fgets(line, sizeof(line), stdin) != NULL) {
int len = strlen(line);
if (len && line[len - 1] == '\n')
line[--len] = 0;
@@ -2250,6 +2291,31 @@ static void get_object_list(int ac, const char **av)
loosen_unused_packed_objects(&revs);
}
+static void rollback_repack(void)
+{
+ struct strbuf dst = STRBUF_INIT;
+ int i, ret;
+ for (i = 0; i < backup_files.nr; i++) {
+ const char *src = backup_files.items[i].string;
+ strbuf_addstr(&dst, src);
+ strbuf_setlen(&dst, dst.len - 4); /* remove .old */
+ ret = rename(src, dst.buf);
+ if (ret)
+ warning("failed to restore %s: %s", src, strerror(errno));
+ strbuf_setlen(&dst, 0);
+ }
+ strbuf_release(&dst);
+ string_list_clear(&backup_files, 0);
+}
+
+static void rollback_repack_on_signal(int signo)
+{
+ rollback_repack();
+ sigchain_pop(signo);
+ raise(signo);
+}
+
+
static int option_parse_index_version(const struct option *opt,
const char *arg, int unset)
{
@@ -2267,12 +2333,14 @@ static int option_parse_index_version(const struct option *opt,
int cmd_pack_objects(int argc, const char **argv, const char *prefix)
{
+ struct strbuf repack_base_name = STRBUF_INIT;
int use_internal_rev_list = 0;
int thin = 0;
int all_progress_implied = 0;
const char **rp_av;
int rp_ac_alloc = 64;
int rp_ac;
+ int i;
struct option pack_objects_options[] = {
OPT_SET_INT('q', NULL, &progress,
"do not show progress meter", 0),
@@ -2328,6 +2396,16 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
"pack compression level"),
OPT_SET_INT(0, "keep-true-parents", &grafts_replace_parents,
"do not hide commits by grafts", 0),
+
+ OPT_BIT(0, "repack", &repack_flags,
+ "repack mode", REPACK_IN_PROGRESS),
+ OPT_BIT(0, "repack-all", &repack_flags,
+ "repack everything into one pack", REPACK_ALL_INTO_ONE),
+ OPT_BIT(0, "remove-redundant", &repack_flags,
+ "remove redundant objects after repack", REPACK_REMOVE_REDUNDANT),
+ OPT_BIT(0, "update-info", &repack_flags,
+ "run git-update-server-info after repack", REPACK_UPDATE_INFO),
+
OPT_END(),
};
@@ -2374,6 +2452,39 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
if (delta_search_threads != 1)
warning("no threads support, ignoring %s", arg);
#endif
+ if ((repack_flags & REPACK_IN_PROGRESS) == 0 &&
+ (repack_flags & ~REPACK_IN_PROGRESS))
+ die("--repack must be given for any repack related options");
+ if (repack_flags & REPACK_IN_PROGRESS) {
+ if (pack_to_stdout)
+ die("--stdout cannot be used with --repack");
+ if (argc)
+ die("base name cannot be used with --repack");
+
+ if (rp_ac + 2 >= rp_ac_alloc) {
+ rp_ac_alloc = alloc_nr(rp_ac_alloc);
+ rp_av = xrealloc(rp_av, rp_ac_alloc * sizeof(*rp_av));
+ }
+ rp_av[rp_ac++] = "--all";
+ rp_av[rp_ac++] = "--reflog";
+ use_internal_rev_list = 1;
+
+ grafts_replace_parents = 0; /* --keep-true-parents */
+ ignore_packed_keep = 1; /* --honor-pack-keep */
+ non_empty = 1; /* --non-empty */
+
+ if (!(repack_flags & REPACK_ALL_INTO_ONE)) {
+ incremental = 1; /* --incremental */
+ rp_av[rp_ac++] = "--unpacked";
+ }
+
+ strbuf_addf(&repack_base_name,
+ "%s/pack/pack", get_object_directory());
+ base_name = repack_base_name.buf;
+
+ sigchain_push_common(rollback_repack_on_signal);
+ atexit(rollback_repack);
+ }
/* Traditionally "pack-objects [options] base extra" failed;
* we would however want to take refs parameter that would
@@ -2388,7 +2499,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
* walker.
*/
- if (!pack_to_stdout) {
+ else if (!pack_to_stdout) {
if (!argc)
die("base name required if --stdout is not given");
base_name = argv[0];
@@ -2439,5 +2550,185 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
fprintf(stderr, "Total %"PRIu32" (delta %"PRIu32"),"
" reused %"PRIu32" (delta %"PRIu32")\n",
written, written_delta, reused, reused_delta);
+
+ if (!(repack_flags & REPACK_IN_PROGRESS))
+ return 0;
+
+ if (!nr_written_packs) {
+ printf(_("Nothing new to pack.\n"));
+ return 0;
+ }
+
+ /* At this point all new packs should be in place. We can
+ safely remove old ones */
+ for (i = 0; i < backup_files.nr; i++) {
+ const char *s = backup_files.items[i].string;
+ int ret = unlink(s);
+ if (ret)
+ warning("failed to remove %s: %s", s, strerror(errno));
+ }
+ string_list_clear(&backup_files, 0);
+
+ if (repack_flags & REPACK_REMOVE_REDUNDANT) {
+ struct packed_git *p;
+ struct string_list to_be_removed = STRING_LIST_INIT_DUP;
+
+ /* free_pack_by_name() may have freed a few packs in
+ write_pack_file() */
+ reprepare_packed_git();
+ for (p = packed_git; p; p = p->next) {
+ if (!p->pack_local || p->pack_keep)
+ continue;
+
+ for (i = 0; i < written_packs.nr; i++) {
+ char *s = written_packs.items[i].string;
+ if (!strcmp(s, p->pack_name))
+ break;
+ }
+ if (i < written_packs.nr)
+ continue;
+
+ string_list_append(&to_be_removed, p->pack_name);
+ }
+ written_packs.strdup_strings = 1;
+ string_list_clear(&written_packs, 0);
+
+ for (i = 0; i < to_be_removed.nr; i++) {
+ char *path = to_be_removed.items[i].string;
+
+ /* Windows limitation on unlink().
+ See c74faea19e39ca933492f697596310397175c329 */
+ free_pack_by_name(path);
+
+ if (unlink(path))
+ warning("failed to remove %s: %s", path, strerror(errno));
+ strcpy(path + strlen(path)-5, ".idx");
+ if (unlink(path))
+ warning("failed to remove %s: %s", path, strerror(errno));
+ }
+ string_list_clear(&to_be_removed, 0);
+
+ reprepare_packed_git();
+ prune_packed_objects(0);
+ }
+
+ if (repack_flags & REPACK_UPDATE_INFO)
+ update_server_info(0);
+
return 0;
}
+
+static int repack_config(const char *k, const char *v, void *cb)
+{
+ if (!strcasecmp(k, "repack.usedeltabaseoffset")) {
+ repack_usedeltabaseoffset = git_config_bool(k, v);
+ return 0;
+ }
+ return git_default_config(k, v, cb);
+}
+
+int cmd_repack(int argc, const char **argv, const char *prefix)
+{
+ int all_in_one = 0;
+ int all_in_one_and_unreachable = 0;
+ int unpack_unreachable = 0;
+ int remove_redundant = 0;
+ int no_reuse_delta = 0;
+ int no_reuse_object = 0;
+ int no_update = 0;
+ int quiet = 0;
+ int local = 0;
+
+ struct option opts[] = {
+ OPT_BOOL('a', NULL, &all_in_one,
+ "pack everything in a single pack"),
+ OPT_BOOL('A', NULL, &all_in_one_and_unreachable,
+ "same as -a, and turn unreachable objects loose"),
+ OPT_BOOL('d', NULL, &remove_redundant,
+ "remove redundant packs, and run git-prune-packed"),
+ OPT_BOOL('f', NULL, &no_reuse_delta,
+ "pass --no-reuse-delta to git-pack-objects"),
+ OPT_BOOL('F', NULL, &no_reuse_object,
+ "pass --no-reuse-object to git-pack-objects"),
+ OPT_BOOL('n', NULL, &no_update,
+ "do not run git-update-server-info"),
+ OPT_BOOL('q', NULL, &quiet, "be quiet"),
+ OPT_BOOL('l', NULL, &local,
+ "pass --local to git-pack-objects"),
+ { OPTION_ARGUMENT, 0, "window", NULL, "n",
+ "size of the window used for delta compression", 0 },
+ { OPTION_ARGUMENT, 0, "window-memory", NULL, "n",
+ "same as the above, but limit memory size instead of entries count", 0 },
+ { OPTION_ARGUMENT, 0, "depth", NULL, "n",
+ "limits the maximum delta depth", 0 },
+ { OPTION_ARGUMENT, 0, "max-pack-size", NULL, "n",
+ "maximum size of each packfile", 0},
+ OPT_END(),
+ };
+
+ const char *av[] = { "pack-objects", "--repack",
+ NULL, /* --[no-]update-info */
+ NULL, /* --delta-base-offset */
+ NULL, /* --repack-all */
+ NULL, /* --remove-redundant */
+ NULL, /* --no-reuse-delta */
+ NULL, /* --no-reuse-object */
+ NULL, /* --local */
+ NULL, /* -q */
+ NULL, /* --unpack-unreachable */
+ NULL, /* --window */
+ NULL, /* --window-memory */
+ NULL, /* --depth */
+ NULL, /* --max-pack-size */
+ NULL
+ };
+ int ac = 2;
+
+ git_config(repack_config, NULL);
+
+ argc = parse_options(argc, argv, prefix, opts, repack_usage, 0);
+
+ if (no_update)
+ av[ac++] = "--no-update-info";
+ else
+ av[ac++] = "--update-info";
+ if (repack_usedeltabaseoffset)
+ av[ac++] = "--delta-base-offset";
+ if (all_in_one_and_unreachable) {
+ av[ac++] = "--repack-all";
+ all_in_one = 1;
+ unpack_unreachable = 1;
+ }
+ if (all_in_one)
+ av[ac++] = "--repack-all";
+ if (remove_redundant)
+ av[ac++] = "--remove-redundant";
+ if (no_reuse_delta)
+ av[ac++] = "--no-reuse-delta";
+ if (no_reuse_object)
+ av[ac++] = "--no-reuse-object";
+ if (local)
+ av[ac++] = "--local";
+ if (quiet)
+ av[ac++] = "-q";
+ if ((ac + argc) * sizeof(*av) > sizeof(av))
+ die("Too many options");
+ memcpy(av + ac, argv, argc * sizeof(*argv));
+ ac += argc;
+
+ if (all_in_one && remove_redundant) {
+ struct packed_git *p;
+
+ prepare_packed_git();
+ for (p = packed_git; p; p = p->next) {
+ if (!p->pack_keep &&
+ unpack_unreachable && remove_redundant) {
+ av[ac++] = "--unpack-unreachable";
+ break;
+ }
+ }
+ }
+
+ trace_argv_printf(av, "trace: built-in: git");
+ return cmd_pack_objects(ac, av, prefix);
+}
diff --git a/contrib/examples/git-repack.sh b/contrib/examples/git-repack.sh
new file mode 100755
index 0000000..624feec
--- /dev/null
+++ b/contrib/examples/git-repack.sh
@@ -0,0 +1,186 @@
+#!/bin/sh
+#
+# Copyright (c) 2005 Linus Torvalds
+#
+
+OPTIONS_KEEPDASHDASH=
+OPTIONS_SPEC="\
+git repack [options]
+--
+a pack everything in a single pack
+A same as -a, and turn unreachable objects loose
+d remove redundant packs, and run git-prune-packed
+f pass --no-reuse-delta to git-pack-objects
+F pass --no-reuse-object to git-pack-objects
+n do not run git-update-server-info
+q,quiet be quiet
+l pass --local to git-pack-objects
+ Packing constraints
+window= size of the window used for delta compression
+window-memory= same as the above, but limit memory size instead of entries count
+depth= limits the maximum delta depth
+max-pack-size= maximum size of each packfile
+"
+SUBDIRECTORY_OK='Yes'
+. git-sh-setup
+
+no_update_info= all_into_one= remove_redundant= unpack_unreachable=
+local= no_reuse= extra=
+while test $# != 0
+do
+ case "$1" in
+ -n) no_update_info=t ;;
+ -a) all_into_one=t ;;
+ -A) all_into_one=t
+ unpack_unreachable=--unpack-unreachable ;;
+ -d) remove_redundant=t ;;
+ -q) GIT_QUIET=t ;;
+ -f) no_reuse=--no-reuse-delta ;;
+ -F) no_reuse=--no-reuse-object ;;
+ -l) local=--local ;;
+ --max-pack-size|--window|--window-memory|--depth)
+ extra="$extra $1=$2"; shift ;;
+ --) shift; break;;
+ *) usage ;;
+ esac
+ shift
+done
+
+case "`git config --bool repack.usedeltabaseoffset || echo true`" in
+true)
+ extra="$extra --delta-base-offset" ;;
+esac
+
+PACKDIR="$GIT_OBJECT_DIRECTORY/pack"
+PACKTMP="$PACKDIR/.tmp-$$-pack"
+rm -f "$PACKTMP"-*
+trap 'rm -f "$PACKTMP"-*' 0 1 2 3 15
+
+# There will be more repacking strategies to come...
+case ",$all_into_one," in
+,,)
+ args='--unpacked --incremental'
+ ;;
+,t,)
+ args= existing=
+ if [ -d "$PACKDIR" ]; then
+ for e in `cd "$PACKDIR" && find . -type f -name '*.pack' \
+ | sed -e 's/^\.\///' -e 's/\.pack$//'`
+ do
+ if [ -e "$PACKDIR/$e.keep" ]; then
+ : keep
+ else
+ existing="$existing $e"
+ fi
+ done
+ if test -n "$existing" -a -n "$unpack_unreachable" -a \
+ -n "$remove_redundant"
+ then
+ args="$args $unpack_unreachable"
+ fi
+ fi
+ ;;
+esac
+
+mkdir -p "$PACKDIR" || exit
+
+args="$args $local ${GIT_QUIET:+-q} $no_reuse$extra"
+names=$(git pack-objects --keep-true-parents --honor-pack-keep --non-empty --all --reflog $args </dev/null "$PACKTMP") ||
+ exit 1
+if [ -z "$names" ]; then
+ say Nothing new to pack.
+fi
+
+# Ok we have prepared all new packfiles.
+
+# First see if there are packs of the same name and if so
+# if we can move them out of the way (this can happen if we
+# repacked immediately after packing fully.
+rollback=
+failed=
+for name in $names
+do
+ for sfx in pack idx
+ do
+ file=pack-$name.$sfx
+ test -f "$PACKDIR/$file" || continue
+ rm -f "$PACKDIR/old-$file" &&
+ mv "$PACKDIR/$file" "$PACKDIR/old-$file" || {
+ failed=t
+ break
+ }
+ rollback="$rollback $file"
+ done
+ test -z "$failed" || break
+done
+
+# If renaming failed for any of them, roll the ones we have
+# already renamed back to their original names.
+if test -n "$failed"
+then
+ rollback_failure=
+ for file in $rollback
+ do
+ mv "$PACKDIR/old-$file" "$PACKDIR/$file" ||
+ rollback_failure="$rollback_failure $file"
+ done
+ if test -n "$rollback_failure"
+ then
+ echo >&2 "WARNING: Some packs in use have been renamed by"
+ echo >&2 "WARNING: prefixing old- to their name, in order to"
+ echo >&2 "WARNING: replace them with the new version of the"
+ echo >&2 "WARNING: file. But the operation failed, and"
+ echo >&2 "WARNING: attempt to rename them back to their"
+ echo >&2 "WARNING: original names also failed."
+ echo >&2 "WARNING: Please rename them in $PACKDIR manually:"
+ for file in $rollback_failure
+ do
+ echo >&2 "WARNING: old-$file -> $file"
+ done
+ fi
+ exit 1
+fi
+
+# Now the ones with the same name are out of the way...
+fullbases=
+for name in $names
+do
+ fullbases="$fullbases pack-$name"
+ chmod a-w "$PACKTMP-$name.pack"
+ chmod a-w "$PACKTMP-$name.idx"
+ mv -f "$PACKTMP-$name.pack" "$PACKDIR/pack-$name.pack" &&
+ mv -f "$PACKTMP-$name.idx" "$PACKDIR/pack-$name.idx" ||
+ exit
+done
+
+# Remove the "old-" files
+for name in $names
+do
+ rm -f "$PACKDIR/old-pack-$name.idx"
+ rm -f "$PACKDIR/old-pack-$name.pack"
+done
+
+# End of pack replacement.
+
+if test "$remove_redundant" = t
+then
+ # We know $existing are all redundant.
+ if [ -n "$existing" ]
+ then
+ ( cd "$PACKDIR" &&
+ for e in $existing
+ do
+ case " $fullbases " in
+ *" $e "*) ;;
+ *) rm -f "$e.pack" "$e.idx" "$e.keep" ;;
+ esac
+ done
+ )
+ fi
+ git prune-packed ${GIT_QUIET:+-q}
+fi
+
+case "$no_update_info" in
+t) : ;;
+*) git update-server-info ;;
+esac
diff --git a/git-repack.sh b/git-repack.sh
deleted file mode 100755
index 624feec..0000000
--- a/git-repack.sh
+++ /dev/null
@@ -1,186 +0,0 @@
-#!/bin/sh
-#
-# Copyright (c) 2005 Linus Torvalds
-#
-
-OPTIONS_KEEPDASHDASH=
-OPTIONS_SPEC="\
-git repack [options]
---
-a pack everything in a single pack
-A same as -a, and turn unreachable objects loose
-d remove redundant packs, and run git-prune-packed
-f pass --no-reuse-delta to git-pack-objects
-F pass --no-reuse-object to git-pack-objects
-n do not run git-update-server-info
-q,quiet be quiet
-l pass --local to git-pack-objects
- Packing constraints
-window= size of the window used for delta compression
-window-memory= same as the above, but limit memory size instead of entries count
-depth= limits the maximum delta depth
-max-pack-size= maximum size of each packfile
-"
-SUBDIRECTORY_OK='Yes'
-. git-sh-setup
-
-no_update_info= all_into_one= remove_redundant= unpack_unreachable=
-local= no_reuse= extra=
-while test $# != 0
-do
- case "$1" in
- -n) no_update_info=t ;;
- -a) all_into_one=t ;;
- -A) all_into_one=t
- unpack_unreachable=--unpack-unreachable ;;
- -d) remove_redundant=t ;;
- -q) GIT_QUIET=t ;;
- -f) no_reuse=--no-reuse-delta ;;
- -F) no_reuse=--no-reuse-object ;;
- -l) local=--local ;;
- --max-pack-size|--window|--window-memory|--depth)
- extra="$extra $1=$2"; shift ;;
- --) shift; break;;
- *) usage ;;
- esac
- shift
-done
-
-case "`git config --bool repack.usedeltabaseoffset || echo true`" in
-true)
- extra="$extra --delta-base-offset" ;;
-esac
-
-PACKDIR="$GIT_OBJECT_DIRECTORY/pack"
-PACKTMP="$PACKDIR/.tmp-$$-pack"
-rm -f "$PACKTMP"-*
-trap 'rm -f "$PACKTMP"-*' 0 1 2 3 15
-
-# There will be more repacking strategies to come...
-case ",$all_into_one," in
-,,)
- args='--unpacked --incremental'
- ;;
-,t,)
- args= existing=
- if [ -d "$PACKDIR" ]; then
- for e in `cd "$PACKDIR" && find . -type f -name '*.pack' \
- | sed -e 's/^\.\///' -e 's/\.pack$//'`
- do
- if [ -e "$PACKDIR/$e.keep" ]; then
- : keep
- else
- existing="$existing $e"
- fi
- done
- if test -n "$existing" -a -n "$unpack_unreachable" -a \
- -n "$remove_redundant"
- then
- args="$args $unpack_unreachable"
- fi
- fi
- ;;
-esac
-
-mkdir -p "$PACKDIR" || exit
-
-args="$args $local ${GIT_QUIET:+-q} $no_reuse$extra"
-names=$(git pack-objects --keep-true-parents --honor-pack-keep --non-empty --all --reflog $args </dev/null "$PACKTMP") ||
- exit 1
-if [ -z "$names" ]; then
- say Nothing new to pack.
-fi
-
-# Ok we have prepared all new packfiles.
-
-# First see if there are packs of the same name and if so
-# if we can move them out of the way (this can happen if we
-# repacked immediately after packing fully.
-rollback=
-failed=
-for name in $names
-do
- for sfx in pack idx
- do
- file=pack-$name.$sfx
- test -f "$PACKDIR/$file" || continue
- rm -f "$PACKDIR/old-$file" &&
- mv "$PACKDIR/$file" "$PACKDIR/old-$file" || {
- failed=t
- break
- }
- rollback="$rollback $file"
- done
- test -z "$failed" || break
-done
-
-# If renaming failed for any of them, roll the ones we have
-# already renamed back to their original names.
-if test -n "$failed"
-then
- rollback_failure=
- for file in $rollback
- do
- mv "$PACKDIR/old-$file" "$PACKDIR/$file" ||
- rollback_failure="$rollback_failure $file"
- done
- if test -n "$rollback_failure"
- then
- echo >&2 "WARNING: Some packs in use have been renamed by"
- echo >&2 "WARNING: prefixing old- to their name, in order to"
- echo >&2 "WARNING: replace them with the new version of the"
- echo >&2 "WARNING: file. But the operation failed, and"
- echo >&2 "WARNING: attempt to rename them back to their"
- echo >&2 "WARNING: original names also failed."
- echo >&2 "WARNING: Please rename them in $PACKDIR manually:"
- for file in $rollback_failure
- do
- echo >&2 "WARNING: old-$file -> $file"
- done
- fi
- exit 1
-fi
-
-# Now the ones with the same name are out of the way...
-fullbases=
-for name in $names
-do
- fullbases="$fullbases pack-$name"
- chmod a-w "$PACKTMP-$name.pack"
- chmod a-w "$PACKTMP-$name.idx"
- mv -f "$PACKTMP-$name.pack" "$PACKDIR/pack-$name.pack" &&
- mv -f "$PACKTMP-$name.idx" "$PACKDIR/pack-$name.idx" ||
- exit
-done
-
-# Remove the "old-" files
-for name in $names
-do
- rm -f "$PACKDIR/old-pack-$name.idx"
- rm -f "$PACKDIR/old-pack-$name.pack"
-done
-
-# End of pack replacement.
-
-if test "$remove_redundant" = t
-then
- # We know $existing are all redundant.
- if [ -n "$existing" ]
- then
- ( cd "$PACKDIR" &&
- for e in $existing
- do
- case " $fullbases " in
- *" $e "*) ;;
- *) rm -f "$e.pack" "$e.idx" "$e.keep" ;;
- esac
- done
- )
- fi
- git prune-packed ${GIT_QUIET:+-q}
-fi
-
-case "$no_update_info" in
-t) : ;;
-*) git update-server-info ;;
-esac
diff --git a/git.c b/git.c
index 8e34903..05235ca 100644
--- a/git.c
+++ b/git.c
@@ -410,6 +410,7 @@ static void handle_internal_command(int argc, const char **argv)
{ "remote-ext", cmd_remote_ext },
{ "remote-fd", cmd_remote_fd },
{ "replace", cmd_replace, RUN_SETUP },
+ { "repack", cmd_repack, RUN_SETUP },
{ "repo-config", cmd_repo_config, RUN_SETUP_GENTLY },
{ "rerere", cmd_rerere, RUN_SETUP },
{ "reset", cmd_reset, RUN_SETUP },
--
1.7.4.74.g639db
^ permalink raw reply related
* [PATCH 0/14] resumable network bundles
From: Jeff King @ 2011-11-10 7:43 UTC (permalink / raw)
To: git
One possible option for resumable clones that has been discussed is
letting the server point the client by http to a static bundle
containing most of history, followed by a fetch from the actual git repo
(which should be much cheaper now that we have all of the bundled
history). This series implements "step 0" of this plan: just letting
bundles be fetched across the network in the first place.
Shawn raised some issues about using bundles for this (as opposed to
accessing the packfiles themselves); specifically, this raises the I/O
footprint of a repository that has to serve both the bundled version of
the pack and the regular packfile.
So it may be that we don't follow this plan all the way through.
However, even if we don't, fetching bundles over http is still a useful
thing to be able to do. Which makes this first step worth doing either
way.
[01/14]: t/lib-httpd: check for NO_CURL
[02/14]: http: turn off curl signals
[03/14]: http: refactor http_request function
[04/14]: http: add a public function for arbitrary-callback request
[05/14]: remote-curl: use http callback for requesting refs
[06/14]: transport: factor out bundle to ref list conversion
[07/14]: bundle: add is_bundle_buf helper
[08/14]: remote-curl: free "discovery" object
[09/14]: remote-curl: auto-detect bundles when fetching refs
[10/14]: remote-curl: try base $URL after $URL/info/refs
[11/14]: progress: allow pure-throughput progress meters
[12/14]: remote-curl: show progress for bundle downloads
[13/14]: remote-curl: resume interrupted bundle transfers
[14/14]: clone: give advice on how to resume a failed clone
-Peff
^ permalink raw reply
* Re: [PATCH 0/14] resumable network bundles
From: Jeff King @ 2011-11-10 7:45 UTC (permalink / raw)
To: git
In-Reply-To: <20111110074330.GA27925@sigill.intra.peff.net>
On Thu, Nov 10, 2011 at 02:43:30AM -0500, Jeff King wrote:
> [01/14]: t/lib-httpd: check for NO_CURL
> [02/14]: http: turn off curl signals
> [03/14]: http: refactor http_request function
> [04/14]: http: add a public function for arbitrary-callback request
> [05/14]: remote-curl: use http callback for requesting refs
> [06/14]: transport: factor out bundle to ref list conversion
> [07/14]: bundle: add is_bundle_buf helper
> [08/14]: remote-curl: free "discovery" object
> [09/14]: remote-curl: auto-detect bundles when fetching refs
> [10/14]: remote-curl: try base $URL after $URL/info/refs
> [11/14]: progress: allow pure-throughput progress meters
> [12/14]: remote-curl: show progress for bundle downloads
> [13/14]: remote-curl: resume interrupted bundle transfers
> [14/14]: clone: give advice on how to resume a failed clone
I forgot to mention: this goes on top of mf/curl-select-fdset. It's only
in next now, but some of my http cleanups build semantically on the
cleanups in that topic.
-Peff
^ permalink raw reply
* [PATCH 01/14] t/lib-httpd: check for NO_CURL
From: Jeff King @ 2011-11-10 7:46 UTC (permalink / raw)
To: git
In-Reply-To: <20111110074330.GA27925@sigill.intra.peff.net>
We already do this check individually in each of the tests
that includes lib-httpd. Let's factor it out.
There is one test (t5540) that uses lib-httpd but does not
currently do this check. But it actually has a stricter
check which is a superset (it needs all of the requirements
to have built git-http-push, one of which is not setting
NO_CURL), so adding this extra check won't hurt anything.
Signed-off-by: Jeff King <peff@peff.net>
---
t/lib-httpd.sh | 5 +++++
t/t5541-http-push.sh | 5 -----
t/t5550-http-fetch.sh | 5 -----
t/t5551-http-fetch.sh | 5 -----
t/t5561-http-backend.sh | 5 -----
5 files changed, 5 insertions(+), 20 deletions(-)
diff --git a/t/lib-httpd.sh b/t/lib-httpd.sh
index f7dc078..8331527 100644
--- a/t/lib-httpd.sh
+++ b/t/lib-httpd.sh
@@ -3,6 +3,11 @@
# Copyright (c) 2008 Clemens Buchacher <drizzd@aon.at>
#
+if test -n "$NO_CURL"; then
+ skip_all='skipping test, git built without http support'
+ test_done
+fi
+
if test -z "$GIT_TEST_HTTPD"
then
skip_all="Network testing disabled (define GIT_TEST_HTTPD to enable)"
diff --git a/t/t5541-http-push.sh b/t/t5541-http-push.sh
index a73c826..a326ee0 100755
--- a/t/t5541-http-push.sh
+++ b/t/t5541-http-push.sh
@@ -6,11 +6,6 @@
test_description='test smart pushing over http via http-backend'
. ./test-lib.sh
-if test -n "$NO_CURL"; then
- skip_all='skipping test, git built without http support'
- test_done
-fi
-
ROOT_PATH="$PWD"
LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5541'}
. "$TEST_DIRECTORY"/lib-httpd.sh
diff --git a/t/t5550-http-fetch.sh b/t/t5550-http-fetch.sh
index 311a33c..e9282c5 100755
--- a/t/t5550-http-fetch.sh
+++ b/t/t5550-http-fetch.sh
@@ -3,11 +3,6 @@
test_description='test dumb fetching over http via static file'
. ./test-lib.sh
-if test -n "$NO_CURL"; then
- skip_all='skipping test, git built without http support'
- test_done
-fi
-
LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5550'}
. "$TEST_DIRECTORY"/lib-httpd.sh
start_httpd
diff --git a/t/t5551-http-fetch.sh b/t/t5551-http-fetch.sh
index 26d3557..3557f2e 100755
--- a/t/t5551-http-fetch.sh
+++ b/t/t5551-http-fetch.sh
@@ -3,11 +3,6 @@
test_description='test smart fetching over http via http-backend'
. ./test-lib.sh
-if test -n "$NO_CURL"; then
- skip_all='skipping test, git built without http support'
- test_done
-fi
-
LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5551'}
. "$TEST_DIRECTORY"/lib-httpd.sh
start_httpd
diff --git a/t/t5561-http-backend.sh b/t/t5561-http-backend.sh
index b5d7fbc..974be7c 100755
--- a/t/t5561-http-backend.sh
+++ b/t/t5561-http-backend.sh
@@ -3,11 +3,6 @@
test_description='test git-http-backend'
. ./test-lib.sh
-if test -n "$NO_CURL"; then
- skip_all='skipping test, git built without http support'
- test_done
-fi
-
LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5561'}
. "$TEST_DIRECTORY"/lib-httpd.sh
start_httpd
--
1.7.7.2.7.g9f96f
^ permalink raw reply related
* [PATCH 02/14] http: turn off curl signals
From: Jeff King @ 2011-11-10 7:48 UTC (permalink / raw)
To: git
In-Reply-To: <20111110074330.GA27925@sigill.intra.peff.net>
Curl sets and clears the handler for SIGALRM, which makes it
incompatible with git's progress code. However, we can ask
curl not to do this.
Signed-off-by: Jeff King <peff@peff.net>
---
I'm a little iffy on this one. If I understand correctly, depending on
the build and configuration, curl may not be able to timeout during DNS
lookups. But I'm not sure if it does, anyway, since we don't set any
timeouts.
An alternate plan would be to give the progress code a mode where it
gets poked by curl every second or so (curl has a PROGRESSFUNCTION
option for doing this).
http.c | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/http.c b/http.c
index 95c2dfd..4f9e004 100644
--- a/http.c
+++ b/http.c
@@ -318,6 +318,8 @@ static int has_cert_password(void)
if (curl_http_proxy)
curl_easy_setopt(result, CURLOPT_PROXY, curl_http_proxy);
+ curl_easy_setopt(result, CURLOPT_NOSIGNAL, 1);
+
return result;
}
--
1.7.7.2.7.g9f96f
^ permalink raw reply related
* [PATCH 03/14] http: refactor http_request function
From: Jeff King @ 2011-11-10 7:48 UTC (permalink / raw)
To: git
In-Reply-To: <20111110074330.GA27925@sigill.intra.peff.net>
This function takes a flag to indicate where the output
should go (either to a file or to a strbuf). This flag is
mostly used to set the callback function we hand to curl.
This isn't very flexible for adding new output types.
Instead, let's just let callers pass in the callback
function directly. This results in shorter, more readable,
and more flexible code.
The only other thing the flag was used for was to set a
"Range" header when we already have a partial file (by using
the results of ftell). This patch also adds an "offset"
parameter, which can be used by callers to specify this
feature separately (which is also more flexible, as non-FILE
callers can now resume partial transfers).
Signed-off-by: Jeff King <peff@peff.net>
---
http.c | 37 ++++++++++++++-----------------------
1 files changed, 14 insertions(+), 23 deletions(-)
diff --git a/http.c b/http.c
index 4f9e004..9ffd894 100644
--- a/http.c
+++ b/http.c
@@ -797,11 +797,8 @@ void append_remote_object_url(struct strbuf *buf, const char *url,
return strbuf_detach(&buf, NULL);
}
-/* http_request() targets */
-#define HTTP_REQUEST_STRBUF 0
-#define HTTP_REQUEST_FILE 1
-
-static int http_request(const char *url, void *result, int target, int options)
+static int http_request(const char *url, curl_write_callback cb, void *result,
+ long offset, int options)
{
struct active_request_slot *slot;
struct slot_results results;
@@ -818,19 +815,13 @@ static int http_request(const char *url, void *result, int target, int options)
} else {
curl_easy_setopt(slot->curl, CURLOPT_NOBODY, 0);
curl_easy_setopt(slot->curl, CURLOPT_FILE, result);
+ curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, cb);
+ }
- if (target == HTTP_REQUEST_FILE) {
- long posn = ftell(result);
- curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION,
- fwrite);
- if (posn > 0) {
- strbuf_addf(&buf, "Range: bytes=%ld-", posn);
- headers = curl_slist_append(headers, buf.buf);
- strbuf_reset(&buf);
- }
- } else
- curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION,
- fwrite_buffer);
+ if (offset > 0) {
+ strbuf_addf(&buf, "Range: bytes=%lu-", offset);
+ headers = curl_slist_append(headers, buf.buf);
+ strbuf_reset(&buf);
}
strbuf_addstr(&buf, "Pragma:");
@@ -881,18 +872,18 @@ static int http_request(const char *url, void *result, int target, int options)
return ret;
}
-static int http_request_reauth(const char *url, void *result, int target,
- int options)
+static int http_request_reauth(const char *url, curl_write_callback cb,
+ void *result, unsigned long offset, int options)
{
- int ret = http_request(url, result, target, options);
+ int ret = http_request(url, cb, result, offset, options);
if (ret != HTTP_REAUTH)
return ret;
- return http_request(url, result, target, options);
+ return http_request(url, cb, result, offset, options);
}
int http_get_strbuf(const char *url, struct strbuf *result, int options)
{
- return http_request_reauth(url, result, HTTP_REQUEST_STRBUF, options);
+ return http_request_reauth(url, fwrite_buffer, result, 0, options);
}
/*
@@ -915,7 +906,7 @@ static int http_get_file(const char *url, const char *filename, int options)
goto cleanup;
}
- ret = http_request_reauth(url, result, HTTP_REQUEST_FILE, options);
+ ret = http_request_reauth(url, NULL, result, ftell(result), options);
fclose(result);
if ((ret == HTTP_OK) && move_temp_to_file(tmpfile.buf, filename))
--
1.7.7.2.7.g9f96f
^ permalink raw reply related
* Re: RFH: unexpected reflog behavior with --since=
From: Eric Raible @ 2011-11-10 7:48 UTC (permalink / raw)
To: Jeff King; +Cc: git@vger.kernel.org
In-Reply-To: <20111109220128.GA31535@sigill.intra.peff.net>
On 11/9/2011 2:01 PM, Jeff King wrote:
> On Tue, Nov 08, 2011 at 04:22:41PM -0800, Eric Raible wrote:
>
> [explanation how --since is used to limits traversal omitted]
Yes, all that is as expected, and makes sense.
> Now let's look at reflog walking. It's kind of bolted on to the side
> of the revision traversal machinery. We walk through the reflog
> backwards and pretend that entry N's parent is entry N-1 (you can see
> this if you do "git log -g -p", for example; you see the patch versus
> the last reflog entry, not the patch against the commit's true parent).
>
> In the case of rewound history (like the reset you showed above), this
> means that the history graph will appear to have bad clock skew. The
> timestamp of HEAD@{0} is going to be much earlier than its pretend
> parent, HEAD@{1}. And the "--since" optimization is going to cut off
> traversal, even though there are more interesting commits to be shown.
>
> So in that sense, I think it's a bug, and we should probably disable the
> exit-early-from-traversal optimization when we're walking reflogs.
Indeed. Seems like a case of an optimization leading to an incorrect result.
> But it may also be a misfeature, because it's not clear what you're
> actually trying to limit by. We have commit timestamps, of course, but
> when we are walking reflogs, we also have reflog timestamps. Did you
> actually want to say "show me all commits in the reflog, in reverse
> reflog order, omitting commits that happened before time t"? Or did you
> really mean "show me the reflog entries that happened before time t,
> regardless of their commit timestamp"?
I meant "show me the reflog entries that happened *since* time t,
regardless of their commit timestamp.
> In the latter case, we would either need a new specifier (like
> "--reflog-since"), or to rewrite the commit timestamp when we rewrite
> the parent pointers.
>
> The latter has a certain elegance to it (we are making a pretend linear
> history graph out of the reflog, so faking the timestamps to be sensible
> and in order is a logical thing to do) but I worry about lying too much
> in the output. Something like "git log -g --format=%cd" would now have
> the fake timestamp in the output. But then, we already show the fake
> parents in the output, so I don't know that this is any worse.
Since -g is asking specifying for the reflog, and since the reflog has
its own timestamps, I would expect that those timestamps be used.
^ permalink raw reply
* [PATCH 04/14] http: add a public function for arbitrary-callback request
From: Jeff King @ 2011-11-10 7:49 UTC (permalink / raw)
To: git
In-Reply-To: <20111110074330.GA27925@sigill.intra.peff.net>
The http_request function recently learned to take arbitrary
callbacks; let's expose this functionality to callers.
Signed-off-by: Jeff King <peff@peff.net>
---
http.c | 6 ++++++
http.h | 3 +++
2 files changed, 9 insertions(+), 0 deletions(-)
diff --git a/http.c b/http.c
index 9ffd894..91451e9 100644
--- a/http.c
+++ b/http.c
@@ -886,6 +886,12 @@ int http_get_strbuf(const char *url, struct strbuf *result, int options)
return http_request_reauth(url, fwrite_buffer, result, 0, options);
}
+int http_get_callback(const char *url, curl_write_callback cb,
+ void *data, long offset, int options)
+{
+ return http_request_reauth(url, cb, data, offset, options);
+}
+
/*
* Downloads an url and stores the result in the given file.
*
diff --git a/http.h b/http.h
index ee16069..4977bde 100644
--- a/http.h
+++ b/http.h
@@ -132,6 +132,9 @@ extern void append_remote_object_url(struct strbuf *buf, const char *url,
*/
int http_get_strbuf(const char *url, struct strbuf *result, int options);
+int http_get_callback(const char *url, curl_write_callback cb, void *data,
+ long offset, int options);
+
/*
* Prints an error message using error() containing url and curl_errorstr,
* and returns ret.
--
1.7.7.2.7.g9f96f
^ permalink raw reply related
* [PATCH 05/14] remote-curl: use http callback for requesting refs
From: Jeff King @ 2011-11-10 7:49 UTC (permalink / raw)
To: git
In-Reply-To: <20111110074330.GA27925@sigill.intra.peff.net>
This should behave identically to the current strbuf code,
but opens up room for us to do more clever things with
bundles in a future patch.
Signed-off-by: Jeff King <peff@peff.net>
---
Obviously it's way more code for the same thing, but future patches will
make the design more clear.
remote-curl.c | 22 ++++++++++++++++++++--
1 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/remote-curl.c b/remote-curl.c
index 0e720ee..fb4d853 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -90,6 +90,24 @@ static void free_discovery(struct discovery *d)
}
}
+struct get_refs_cb_data {
+ struct strbuf *out;
+};
+
+static size_t get_refs_callback(char *buf, size_t sz, size_t n, void *vdata)
+{
+ struct get_refs_cb_data *data = vdata;
+ strbuf_add(data->out, buf, sz * n);
+ return sz * n;
+}
+
+static int get_refs_from_url(const char *url, struct strbuf *out, int options)
+{
+ struct get_refs_cb_data data;
+ data.out = out;
+ return http_get_callback(url, get_refs_callback, &data, 0, options);
+}
+
static struct discovery* discover_refs(const char *service)
{
struct strbuf buffer = STRBUF_INIT;
@@ -112,7 +130,7 @@ static void free_discovery(struct discovery *d)
}
refs_url = strbuf_detach(&buffer, NULL);
- http_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);
+ http_ret = get_refs_from_url(refs_url, &buffer, HTTP_NO_CACHE);
/* try again with "plain" url (no ? or & appended) */
if (http_ret != HTTP_OK && http_ret != HTTP_NOAUTH) {
@@ -123,7 +141,7 @@ static void free_discovery(struct discovery *d)
strbuf_addf(&buffer, "%sinfo/refs", url);
refs_url = strbuf_detach(&buffer, NULL);
- http_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);
+ http_ret = get_refs_from_url(refs_url, &buffer, HTTP_NO_CACHE);
}
switch (http_ret) {
--
1.7.7.2.7.g9f96f
^ permalink raw reply related
* [PATCH 06/14] transport: factor out bundle to ref list conversion
From: Jeff King @ 2011-11-10 7:49 UTC (permalink / raw)
To: git
In-Reply-To: <20111110074330.GA27925@sigill.intra.peff.net>
There is a data structure mismatch between what the
transport code wants (a linked list of "struct ref") and
what the bundle header provides (an array of ref names and
sha1s), so the transport code has to convert.
Let's factor out this conversion to make it useful to other
transport-ish callers (like remote-curl).
Signed-off-by: Jeff King <peff@peff.net>
---
bundle.c | 16 ++++++++++++++++
bundle.h | 2 ++
transport.c | 11 +----------
3 files changed, 19 insertions(+), 10 deletions(-)
diff --git a/bundle.c b/bundle.c
index 08020bc..e48fe2f 100644
--- a/bundle.c
+++ b/bundle.c
@@ -7,6 +7,7 @@
#include "list-objects.h"
#include "run-command.h"
#include "refs.h"
+#include "remote.h"
static const char bundle_signature[] = "# v2 git bundle\n";
@@ -449,3 +450,18 @@ int unbundle(struct bundle_header *header, int bundle_fd, int flags)
return error("index-pack died");
return 0;
}
+
+struct ref *bundle_header_to_refs(const struct bundle_header *header)
+{
+ struct ref *result = NULL;
+ int i;
+
+ for (i = 0; i < header->references.nr; i++) {
+ struct ref_list_entry *e = header->references.list + i;
+ struct ref *ref = alloc_ref(e->name);
+ hashcpy(ref->old_sha1, e->sha1);
+ ref->next = result;
+ result = ref;
+ }
+ return result;
+}
diff --git a/bundle.h b/bundle.h
index 1584e4d..675cc97 100644
--- a/bundle.h
+++ b/bundle.h
@@ -24,4 +24,6 @@ int create_bundle(struct bundle_header *header, const char *path,
int list_bundle_refs(struct bundle_header *header,
int argc, const char **argv);
+struct ref *bundle_header_to_refs(const struct bundle_header *header);
+
#endif
diff --git a/transport.c b/transport.c
index 51814b5..5020bbb 100644
--- a/transport.c
+++ b/transport.c
@@ -407,8 +407,6 @@ struct bundle_transport_data {
static struct ref *get_refs_from_bundle(struct transport *transport, int for_push)
{
struct bundle_transport_data *data = transport->data;
- struct ref *result = NULL;
- int i;
if (for_push)
return NULL;
@@ -418,14 +416,7 @@ struct bundle_transport_data {
data->fd = read_bundle_header(transport->url, &data->header);
if (data->fd < 0)
die ("Could not read bundle '%s'.", transport->url);
- for (i = 0; i < data->header.references.nr; i++) {
- struct ref_list_entry *e = data->header.references.list + i;
- struct ref *ref = alloc_ref(e->name);
- hashcpy(ref->old_sha1, e->sha1);
- ref->next = result;
- result = ref;
- }
- return result;
+ return bundle_header_to_refs(&data->header);
}
static int fetch_refs_from_bundle(struct transport *transport,
--
1.7.7.2.7.g9f96f
^ permalink raw reply related
* [PATCH 07/14] bundle: add is_bundle_buf helper
From: Jeff King @ 2011-11-10 7:50 UTC (permalink / raw)
To: git
In-Reply-To: <20111110074330.GA27925@sigill.intra.peff.net>
This is similar to is_bundle, but checks an in-memory buffer
rather than a file. It also works on a partial buffer,
checking only the bundle signature. This means the result is
a three-way conditional: yes, no, or "we do not have enough
data yet".
Signed-off-by: Jeff King <peff@peff.net>
---
bundle.c | 14 ++++++++++++++
bundle.h | 1 +
2 files changed, 15 insertions(+), 0 deletions(-)
diff --git a/bundle.c b/bundle.c
index e48fe2f..6f911bc 100644
--- a/bundle.c
+++ b/bundle.c
@@ -122,6 +122,20 @@ int is_bundle(const char *path, int quiet)
return (fd >= 0);
}
+int is_bundle_buf(const char *s, int len)
+{
+ if (len > strlen(bundle_signature))
+ len = strlen(bundle_signature);
+ /* If we don't match what we already have, then definitely not. */
+ if (memcmp(s, bundle_signature, len))
+ return 0;
+ /* If we have enough bytes, we can say yes */
+ if (len == strlen(bundle_signature))
+ return 1;
+ /* otherwise, we can only say "maybe" */
+ return -1;
+}
+
static int list_refs(struct ref_list *r, int argc, const char **argv)
{
int i;
diff --git a/bundle.h b/bundle.h
index 675cc97..8bec44d 100644
--- a/bundle.h
+++ b/bundle.h
@@ -15,6 +15,7 @@ struct bundle_header {
};
int is_bundle(const char *path, int quiet);
+int is_bundle_buf(const char *s, int len);
int read_bundle_header(const char *path, struct bundle_header *header);
int create_bundle(struct bundle_header *header, const char *path,
int argc, const char **argv);
--
1.7.7.2.7.g9f96f
^ permalink raw reply related
* [PATCH 08/14] remote-curl: free "discovery" object
From: Jeff King @ 2011-11-10 7:50 UTC (permalink / raw)
To: git
In-Reply-To: <20111110074330.GA27925@sigill.intra.peff.net>
We cache the last-used "discovery" object, which contains
the data we pulled from the remote about which refs it has,
which saves us an HTTP round-trip when somebody does
something like "list" followed by "fetch".
We don't bother free()ing it at the end of the program
because it just contains memory which will be reclaimed by
the OS. However, cleaning up explicitly will future-proof us
against later changes which will add external storage (like
temporary files).
Signed-off-by: Jeff King <peff@peff.net>
---
remote-curl.c | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/remote-curl.c b/remote-curl.c
index fb4d853..014d413 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -933,6 +933,7 @@ int main(int argc, const char **argv)
strbuf_reset(&buf);
} while (1);
+ free_discovery(last_discovery);
http_cleanup();
return 0;
--
1.7.7.2.7.g9f96f
^ permalink raw reply related
* [PATCH 09/14] remote-curl: auto-detect bundles when fetching refs
From: Jeff King @ 2011-11-10 7:50 UTC (permalink / raw)
To: git
In-Reply-To: <20111110074330.GA27925@sigill.intra.peff.net>
You can't currently fetch from a network bundle, like:
git fetch http://example.com/foo.bundle
This patch takes the first (and biggest) step towards that
working: it auto-detects when fetching refs results in a
bundle, and automatically spools the bundle to disk and
fetches from it.
There are a few important design decisions to note:
1. We auto-detect the bundle based on content, not based
on a special token in the URL (like ending in
".bundle"). This lets the server side be flexible with
its URLs (e.g., "http://example.com/bundle?repo=foo").
2. When fetching refs, we don't actually fetch $URL, but
start with $URL/info/refs, looking for smart or dumb
http. Some servers, when file "foo.bundle" exists, will
serve it to the client when "foo.bundle/info/refs" is
requested. Therefore we may be "surprised" to receive a
bundle when we thought we were just getting the list of
refs, and need to handle it appropriately.
3. We spool the bundle to disk, and then run "index-pack
--fix-thin" to create a packfile. That means we will
momentarily use twice the size of the bundle in local
disk space. Avoiding this would mean piping directly to
"index-pack --fix-thin". However, if we want to be
able to resume the transfer of the bundle after an
interruption, then we need to save the bundle's pack.
In theory a smart index-pack that was interrupted could
write out its partial results along with a count of how
many bytes it actually consumed (i.e., where to resume
next time), and then pick up where it left off when fed
the rest of the data. But index-pack isn't that smart
yet, so let's start off with spooling.
No tests yet, as apache is not one of the "surprising"
servers from (2), and our test harness is based around that
(though just with this patch, you can fetch from surprising
servers like lighttpd).
Signed-off-by: Jeff King <peff@peff.net>
---
This is really the big, interesting one.
remote-curl.c | 124 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
1 files changed, 119 insertions(+), 5 deletions(-)
diff --git a/remote-curl.c b/remote-curl.c
index 014d413..84586e0 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -7,6 +7,7 @@
#include "run-command.h"
#include "pkt-line.h"
#include "sideband.h"
+#include "bundle.h"
static struct remote *remote;
static const char *url; /* always ends with a trailing slash */
@@ -77,6 +78,10 @@ struct discovery {
char *buf;
size_t len;
unsigned proto_git : 1;
+
+ char *bundle_filename;
+ int bundle_fd;
+ struct bundle_header bundle_header;
};
static struct discovery *last_discovery;
@@ -86,26 +91,93 @@ static void free_discovery(struct discovery *d)
if (d == last_discovery)
last_discovery = NULL;
free(d->buf_alloc);
+ if (d->bundle_fd >= 0)
+ close(d->bundle_fd);
+ if (d->bundle_filename) {
+ unlink(d->bundle_filename);
+ free(d->bundle_filename);
+ }
free(d);
}
}
struct get_refs_cb_data {
struct strbuf *out;
+
+ int is_bundle;
+ const char *tmpname;
+ FILE *fh;
};
static size_t get_refs_callback(char *buf, size_t sz, size_t n, void *vdata)
{
struct get_refs_cb_data *data = vdata;
- strbuf_add(data->out, buf, sz * n);
+ struct strbuf *out = data->out;
+
+ if (data->is_bundle > 0)
+ return fwrite(buf, sz, n, data->fh);
+
+ strbuf_add(out, buf, sz * n);
+
+ if (data->is_bundle == 0)
+ return sz * n;
+
+ data->is_bundle = is_bundle_buf(out->buf, out->len);
+ if (data->is_bundle > 0) {
+ data->fh = fopen(data->tmpname, "wb");
+ if (!data->fh)
+ die_errno("unable to open %s", data->tmpname);
+ if (fwrite(out->buf, 1, out->len, data->fh) < out->len)
+ die_errno("unable to write to %s", data->tmpname);
+ }
return sz * n;
}
-static int get_refs_from_url(const char *url, struct strbuf *out, int options)
+static int get_refs_from_url(const char *url, struct strbuf *out, int options,
+ const char *tmpname, int *is_bundle)
{
struct get_refs_cb_data data;
+ int ret;
+
data.out = out;
- return http_get_callback(url, get_refs_callback, &data, 0, options);
+ data.is_bundle = -1;
+ data.tmpname = tmpname;
+ data.fh = NULL;
+
+ ret = http_get_callback(url, get_refs_callback, &data, 0, options);
+
+ if (data.fh) {
+ if (fclose(data.fh))
+ die_errno("unable to write to %s", data.tmpname);
+ }
+
+ *is_bundle = data.is_bundle > 0;
+ return ret;
+}
+
+static const char *url_to_bundle_tmpfile(const char *url)
+{
+ struct strbuf buf = STRBUF_INIT;
+ int last_was_quoted = 1;
+ const char *ret;
+
+ strbuf_addstr(&buf, "tmp_bundle_");
+ for (; *url; url++) {
+ if (isalpha(*url) || isdigit(*url)) {
+ strbuf_addch(&buf, *url);
+ last_was_quoted = 0;
+ }
+ else if (!last_was_quoted) {
+ strbuf_addch(&buf, '_');
+ last_was_quoted = 1;
+ }
+ }
+ if (last_was_quoted)
+ strbuf_setlen(&buf, buf.len - 1);
+
+ ret = git_path("objects/%s", buf.buf);
+ strbuf_release(&buf);
+ return ret;
}
static struct discovery* discover_refs(const char *service)
@@ -114,11 +186,15 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options)
struct discovery *last = last_discovery;
char *refs_url;
int http_ret, is_http = 0, proto_git_candidate = 1;
+ const char *filename;
+ int is_bundle;
if (last && !strcmp(service, last->service))
return last;
free_discovery(last);
+ filename = url_to_bundle_tmpfile(url);
+
strbuf_addf(&buffer, "%sinfo/refs", url);
if (!prefixcmp(url, "http://") || !prefixcmp(url, "https://")) {
is_http = 1;
@@ -130,7 +206,8 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options)
}
refs_url = strbuf_detach(&buffer, NULL);
- http_ret = get_refs_from_url(refs_url, &buffer, HTTP_NO_CACHE);
+ http_ret = get_refs_from_url(refs_url, &buffer, HTTP_NO_CACHE,
+ filename, &is_bundle);
/* try again with "plain" url (no ? or & appended) */
if (http_ret != HTTP_OK && http_ret != HTTP_NOAUTH) {
@@ -141,7 +218,8 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options)
strbuf_addf(&buffer, "%sinfo/refs", url);
refs_url = strbuf_detach(&buffer, NULL);
- http_ret = get_refs_from_url(refs_url, &buffer, HTTP_NO_CACHE);
+ http_ret = get_refs_from_url(refs_url, &buffer, HTTP_NO_CACHE,
+ filename, &is_bundle);
}
switch (http_ret) {
@@ -161,6 +239,7 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options)
last->service = service;
last->buf_alloc = strbuf_detach(&buffer, &last->len);
last->buf = last->buf_alloc;
+ last->bundle_fd = -1;
if (is_http && proto_git_candidate
&& 5 <= last->len && last->buf[4] == '#') {
@@ -190,6 +269,10 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options)
last->proto_git = 1;
}
+ else if (is_bundle) {
+ last->bundle_filename = xstrdup(filename);
+ }
+
free(refs_url);
strbuf_release(&buffer);
last_discovery = last;
@@ -276,6 +359,22 @@ static int write_discovery(int in, int out, void *data)
return refs;
}
+static void ensure_bundle_open(struct discovery *heads)
+{
+ if (heads->bundle_fd >= 0)
+ return;
+ heads->bundle_fd = read_bundle_header(heads->bundle_filename,
+ &heads->bundle_header);
+ if (heads->bundle_fd < 0)
+ die("could not read bundle from %s", url);
+}
+
+static struct ref *parse_bundle_refs(struct discovery *heads)
+{
+ ensure_bundle_open(heads);
+ return bundle_header_to_refs(&heads->bundle_header);
+}
+
static struct ref *get_refs(int for_push)
{
struct discovery *heads;
@@ -287,6 +386,11 @@ static int write_discovery(int in, int out, void *data)
if (heads->proto_git)
return parse_git_refs(heads);
+ if (heads->bundle_filename) {
+ if (for_push)
+ die("cannot push into a remote bundle");
+ return parse_bundle_refs(heads);
+ }
return parse_info_refs(heads);
}
@@ -690,11 +794,21 @@ static int fetch_git(struct discovery *heads,
return err;
}
+static int fetch_bundle(struct discovery *d,
+ int nr_heads, struct ref **to_fetch)
+{
+ ensure_bundle_open(d);
+ return unbundle(&d->bundle_header, d->bundle_fd,
+ options.progress ? BUNDLE_VERBOSE : 0);
+}
+
static int fetch(int nr_heads, struct ref **to_fetch)
{
struct discovery *d = discover_refs("git-upload-pack");
if (d->proto_git)
return fetch_git(d, nr_heads, to_fetch);
+ else if (d->bundle_filename)
+ return fetch_bundle(d, nr_heads, to_fetch);
else
return fetch_dumb(nr_heads, to_fetch);
}
--
1.7.7.2.7.g9f96f
^ permalink raw reply related
* [PATCH 10/14] remote-curl: try base $URL after $URL/info/refs
From: Jeff King @ 2011-11-10 7:51 UTC (permalink / raw)
To: git
In-Reply-To: <20111110074330.GA27925@sigill.intra.peff.net>
When fetching via http, we will try "$URL/info/refs" to get
the list of refs. We may get an unexpected bundle from that
transfer, and we already handle that case. But we should
also check just "$URL" to see if it's a bundle.
Signed-off-by: Jeff King <peff@peff.net>
---
And now we can actually test with apache.
remote-curl.c | 19 +++++++++++++++++++
t/t5552-http-bundle.sh | 36 ++++++++++++++++++++++++++++++++++++
2 files changed, 55 insertions(+), 0 deletions(-)
create mode 100755 t/t5552-http-bundle.sh
diff --git a/remote-curl.c b/remote-curl.c
index 84586e0..7734495 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -222,6 +222,25 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options,
filename, &is_bundle);
}
+ /* try the straight URL for a bundle, but don't impact the
+ * error reporting that happens below. */
+ if (http_ret != HTTP_OK && http_ret != HTTP_NOAUTH) {
+ struct strbuf trimmed = STRBUF_INIT;
+ int r;
+
+ strbuf_reset(&buffer);
+
+ strbuf_addstr(&trimmed, url);
+ while (trimmed.len > 0 && trimmed.buf[trimmed.len-1] == '/')
+ strbuf_setlen(&trimmed, trimmed.len - 1);
+
+ r = get_refs_from_url(trimmed.buf, &buffer, 0, filename,
+ &is_bundle);
+ if (r == HTTP_OK && is_bundle)
+ http_ret = r;
+ strbuf_release(&trimmed);
+ }
+
switch (http_ret) {
case HTTP_OK:
break;
diff --git a/t/t5552-http-bundle.sh b/t/t5552-http-bundle.sh
new file mode 100755
index 0000000..8527403
--- /dev/null
+++ b/t/t5552-http-bundle.sh
@@ -0,0 +1,36 @@
+#!/bin/sh
+
+test_description='test fetching from http-accessible bundles'
+. ./test-lib.sh
+
+LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5552'}
+. "$TEST_DIRECTORY"/lib-httpd.sh
+start_httpd
+
+test_expect_success 'create bundles' '
+ test_commit one &&
+ git bundle create "$HTTPD_DOCUMENT_ROOT_PATH/one.bundle" --all &&
+ test_commit two &&
+ git bundle create "$HTTPD_DOCUMENT_ROOT_PATH/two.bundle" --all ^one
+'
+
+test_expect_success 'clone from bundle' '
+ git clone --bare $HTTPD_URL/one.bundle clone &&
+ echo one >expect &&
+ git --git-dir=clone log -1 --format=%s >actual &&
+ test_cmp expect actual
+'
+
+test_expect_success 'fetch from bundle' '
+ git --git-dir=clone fetch $HTTPD_URL/two.bundle refs/*:refs/* &&
+ echo two >expect &&
+ git --git-dir=clone log -1 --format=%s >actual &&
+ test_cmp expect actual
+'
+
+test_expect_success 'cannot clone from partial bundle' '
+ test_must_fail git clone $HTTPD_URL/two.bundle
+'
+
+stop_httpd
+test_done
--
1.7.7.2.7.g9f96f
^ permalink raw reply related
* [PATCH 11/14] progress: allow pure-throughput progress meters
From: Jeff King @ 2011-11-10 7:53 UTC (permalink / raw)
To: git
In-Reply-To: <20111110074330.GA27925@sigill.intra.peff.net>
The progress code assumes we are counting something (usually
objects), even if we are measuring throughput. This works
for fetching packfiles, since they show us the object count
alongside the throughput, like:
Receiving objects: 2% (301/11968), 22.00 MiB | 10.97 MiB/s
You can also tell the progress code you don't know how many
items you have (by specifying a total of 0), and it looks
like:
Counting objects: 34957
However, if you're fetching a single large item, you want
throughput but you might not have a meaningful count. You
can say you are getting item 0 or 1 out of 1 total, but then
the percent meter is misleading:
Downloading: 0% (0/1), 22.00 MiB | 10.97 MiB/s
or
Downloading: 100% (0/1), 22.00 MiB | 10.97 MiB/s
Neither of those is accurate. You are probably somewhere
between zero and 100 percent through the operation, but you
don't know how far.
Telling it you don't know how many items is even uglier:
Downloading: 1, 22.00 MiB | 10.97 MiB/s
Instead, this patch will omit the count entirely if you are
on the zero-th item of an unknown number of items. It looks
like:
Downloading: 22.00 MiB | 10.97 MiB/s
Signed-off-by: Jeff King <peff@peff.net>
---
This was the last amount of work to massage the progress code into doing
what I wanted. It might be nicer if it could show a percentage (if we
know the total size), but there's even more surgery required for that.
progress.c | 20 ++++++++++++--------
1 files changed, 12 insertions(+), 8 deletions(-)
diff --git a/progress.c b/progress.c
index 3971f49..92fc3c5 100644
--- a/progress.c
+++ b/progress.c
@@ -103,7 +103,10 @@ static int display(struct progress *progress, unsigned n, const char *done)
return 1;
}
} else if (progress_update) {
- fprintf(stderr, "%s: %u%s%s", progress->title, n, tp, eol);
+ fprintf(stderr, "%s: ", progress->title);
+ if (n)
+ fprintf(stderr, "%u", n);
+ fprintf(stderr, "%s%s", tp, eol);
fflush(stderr);
progress_update = 0;
return 1;
@@ -113,23 +116,24 @@ static int display(struct progress *progress, unsigned n, const char *done)
}
static void throughput_string(struct throughput *tp, off_t total,
- unsigned int rate)
+ unsigned int rate, struct progress *p)
{
+ const char *delim = p->total || p->last_value > 0 ? ", " : "";
int l = sizeof(tp->display);
if (total > 1 << 30) {
- l -= snprintf(tp->display, l, ", %u.%2.2u GiB",
+ l -= snprintf(tp->display, l, "%s%u.%2.2u GiB", delim,
(int)(total >> 30),
(int)(total & ((1 << 30) - 1)) / 10737419);
} else if (total > 1 << 20) {
int x = total + 5243; /* for rounding */
- l -= snprintf(tp->display, l, ", %u.%2.2u MiB",
+ l -= snprintf(tp->display, l, "%s%u.%2.2u MiB", delim,
x >> 20, ((x & ((1 << 20) - 1)) * 100) >> 20);
} else if (total > 1 << 10) {
int x = total + 5; /* for rounding */
- l -= snprintf(tp->display, l, ", %u.%2.2u KiB",
+ l -= snprintf(tp->display, l, "%s%u.%2.2u KiB", delim,
x >> 10, ((x & ((1 << 10) - 1)) * 100) >> 10);
} else {
- l -= snprintf(tp->display, l, ", %u bytes", (int)total);
+ l -= snprintf(tp->display, l, "%s%u bytes", delim, (int)total);
}
if (rate > 1 << 10) {
@@ -197,7 +201,7 @@ void display_throughput(struct progress *progress, off_t total)
tp->last_misecs[tp->idx] = misecs;
tp->idx = (tp->idx + 1) % TP_IDX_MAX;
- throughput_string(tp, total, rate);
+ throughput_string(tp, total, rate, progress);
if (progress->last_value != -1 && progress_update)
display(progress, progress->last_value, NULL);
}
@@ -255,7 +259,7 @@ void stop_progress_msg(struct progress **p_progress, const char *msg)
if (tp) {
unsigned int rate = !tp->avg_misecs ? 0 :
tp->avg_bytes / tp->avg_misecs;
- throughput_string(tp, tp->curr_total, rate);
+ throughput_string(tp, tp->curr_total, rate, progress);
}
progress_update = 1;
sprintf(bufp, ", %s.\n", msg);
--
1.7.7.2.7.g9f96f
^ permalink raw reply related
* [PATCH 12/14] remote-curl: show progress for bundle downloads
From: Jeff King @ 2011-11-10 7:53 UTC (permalink / raw)
To: git
In-Reply-To: <20111110074330.GA27925@sigill.intra.peff.net>
Generally, the point of fetching from a bundle is that it's
big. Without a progress meter, git will appear to hang
during the long download.
This patch adds a throughput meter (i.e., just the bytes
transferred and the rate). In the long run, we should look
for a content-length header from the server so we can show a
total size and completion percentage. However, displaying
that properly will require some surgery to the progress
code, so let's leave it as a future enhancement.
Signed-off-by: Jeff King <peff@peff.net>
---
remote-curl.c | 18 +++++++++++++++++-
1 files changed, 17 insertions(+), 1 deletions(-)
diff --git a/remote-curl.c b/remote-curl.c
index 7734495..6b0820e 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -8,6 +8,7 @@
#include "pkt-line.h"
#include "sideband.h"
#include "bundle.h"
+#include "progress.h"
static struct remote *remote;
static const char *url; /* always ends with a trailing slash */
@@ -107,6 +108,9 @@ struct get_refs_cb_data {
int is_bundle;
const char *tmpname;
FILE *fh;
+
+ struct progress *progress;
+ off_t total;
};
static size_t get_refs_callback(char *buf, size_t sz, size_t n, void *vdata)
@@ -114,8 +118,11 @@ static size_t get_refs_callback(char *buf, size_t sz, size_t n, void *vdata)
struct get_refs_cb_data *data = vdata;
struct strbuf *out = data->out;
- if (data->is_bundle > 0)
+ if (data->is_bundle > 0) {
+ data->total += sz * n;
+ display_throughput(data->progress, data->total);
return fwrite(buf, sz, n, data->fh);
+ }
strbuf_add(out, buf, sz * n);
@@ -129,6 +136,12 @@ static size_t get_refs_callback(char *buf, size_t sz, size_t n, void *vdata)
die_errno("unable to open %s", data->tmpname);
if (fwrite(out->buf, 1, out->len, data->fh) < out->len)
die_errno("unable to write to %s", data->tmpname);
+ if (options.progress) {
+ data->total = out->len;
+ data->progress = start_progress("Downloading bundle", 0);
+ display_progress(data->progress, 0);
+ display_throughput(data->progress, data->total);
+ }
}
return sz * n;
}
@@ -143,6 +156,8 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options,
data.is_bundle = -1;
data.tmpname = tmpname;
data.fh = NULL;
+ data.progress = NULL;
+ data.total = 0;
ret = http_get_callback(url, get_refs_callback, &data, 0, options);
@@ -150,6 +165,7 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options,
if (fclose(data.fh))
die_errno("unable to write to %s", data.tmpname);
}
+ stop_progress(&data.progress);
*is_bundle = data.is_bundle > 0;
return ret;
--
1.7.7.2.7.g9f96f
^ permalink raw reply related
* [PATCH 13/14] remote-curl: resume interrupted bundle transfers
From: Jeff King @ 2011-11-10 7:53 UTC (permalink / raw)
To: git
In-Reply-To: <20111110074330.GA27925@sigill.intra.peff.net>
If we have a bundle file from a previous fetch that matches
this URL, then we should resume the transfer where we left
off.
Signed-off-by: Jeff King <peff@peff.net>
---
The second half of the diff is hard to read because I re-indent a big
chunk. It's much easier to see what's going on with "diff -b".
remote-curl.c | 64 +++++++++++++++++++++++++++++++++++++++++++++++---------
1 files changed, 53 insertions(+), 11 deletions(-)
diff --git a/remote-curl.c b/remote-curl.c
index 6b0820e..43ad1b6 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -9,6 +9,7 @@
#include "sideband.h"
#include "bundle.h"
#include "progress.h"
+#include "dir.h"
static struct remote *remote;
static const char *url; /* always ends with a trailing slash */
@@ -171,6 +172,32 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options,
return ret;
}
+static int resume_bundle(const char *url, const char *tmpname)
+{
+ struct get_refs_cb_data data;
+ int ret;
+
+ data.fh = fopen(tmpname, "ab");
+ if (!data.fh)
+ die_errno("unable to open %s", tmpname);
+
+ data.is_bundle = 1;
+ data.total = ftell(data.fh);
+ if (options.progress) {
+ data.progress = start_progress("Resuming bundle", 0);
+ display_progress(data.progress, 0);
+ display_throughput(data.progress, data.total);
+ }
+
+ ret = http_get_callback(url, get_refs_callback, &data, data.total, 0);
+
+ if (fclose(data.fh))
+ die_errno("unable to write to %s", tmpname);
+ stop_progress(&data.progress);
+
+ return ret;
+}
+
static const char *url_to_bundle_tmpfile(const char *url)
{
struct strbuf buf = STRBUF_INIT;
@@ -210,20 +237,35 @@ static int get_refs_from_url(const char *url, struct strbuf *out, int options,
free_discovery(last);
filename = url_to_bundle_tmpfile(url);
+ if (file_exists(filename)) {
+ struct strbuf trimmed = STRBUF_INIT;
- strbuf_addf(&buffer, "%sinfo/refs", url);
- if (!prefixcmp(url, "http://") || !prefixcmp(url, "https://")) {
- is_http = 1;
- if (!strchr(url, '?'))
- strbuf_addch(&buffer, '?');
- else
- strbuf_addch(&buffer, '&');
- strbuf_addf(&buffer, "service=%s", service);
+ strbuf_addstr(&trimmed, url);
+ while (trimmed.len > 0 && trimmed.buf[trimmed.len-1] == '/')
+ strbuf_setlen(&trimmed, trimmed.len - 1);
+ refs_url = strbuf_detach(&trimmed, NULL);
+
+ http_ret = resume_bundle(refs_url, filename);
+ is_bundle = 1;
}
- refs_url = strbuf_detach(&buffer, NULL);
+ else
+ http_ret = HTTP_MISSING_TARGET;
+
+ if (http_ret != HTTP_OK && http_ret != HTTP_NOAUTH) {
+ strbuf_addf(&buffer, "%sinfo/refs", url);
+ if (!prefixcmp(url, "http://") || !prefixcmp(url, "https://")) {
+ is_http = 1;
+ if (!strchr(url, '?'))
+ strbuf_addch(&buffer, '?');
+ else
+ strbuf_addch(&buffer, '&');
+ strbuf_addf(&buffer, "service=%s", service);
+ }
+ refs_url = strbuf_detach(&buffer, NULL);
- http_ret = get_refs_from_url(refs_url, &buffer, HTTP_NO_CACHE,
- filename, &is_bundle);
+ http_ret = get_refs_from_url(refs_url, &buffer, HTTP_NO_CACHE,
+ filename, &is_bundle);
+ }
/* try again with "plain" url (no ? or & appended) */
if (http_ret != HTTP_OK && http_ret != HTTP_NOAUTH) {
--
1.7.7.2.7.g9f96f
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox