* Re: [RFC] Zit: the git-based single file content tracker
From: Nguyen Thai Ngoc Duy @ 2008-10-23 13:51 UTC (permalink / raw)
To: Giuseppe Bilotta; +Cc: git
In-Reply-To: <cb7bb73a0810230633r9970a50mbb4ecf3a855c3a21@mail.gmail.com>
On 10/23/08, Giuseppe Bilotta <giuseppe.bilotta@gmail.com> wrote:
> On Thu, Oct 23, 2008 at 2:50 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
> > On 10/23/08, Giuseppe Bilotta <giuseppe.bilotta@gmail.com> wrote:
> >> The principle is extremely simple: when you choose to start tracking a
> >> file with Zit,
> >>
> >> zit track file
> >>
> >> Zit will create a directory .zit.file to hold a git repository
> >> tracking the single file .zit.file/file, which is just a hard link to
> >> file.
> >
> > Why not use one .zit repo and track each file on each own branch?.
>
>
> So your proposal is to have a single .zit repo which is actually a git
> repo and where each additional tracked file becomes its own branch,
> and zit would take care of switching from branch to branch when zit
> commands are called?
I don't know if switching is necessary. With one file per pranch, the
index is even not necessary.
> I think this solution would have a number of problems, apart from
> being generally quite messy. First of all, moving a file and its
> history somewhere else means toying around with the history of a much
> wider repo, whereas the current approach would mean just moving the
> .zit.file dir together with the file (modulo hardlinks). Non-linear
> histories for a single file would be more complex to handle, too. And
> publishing just the history of one file would be damn complex.
The history should be linear. Git (or zit) repository is just a
container for git branches. Each branch contains only one file. Moving
a file history is equivalent to "git push" + "git branch -D".
Something like this (not tested):
cd dst
git init
cd src
git push dst local-branch:remote-branch
git branch -D local-branch
git gc
> --
> Giuseppe "Oblomov" Bilotta
>
--
Duy
^ permalink raw reply
* Verifying the whole repository
From: Alex Bennee @ 2008-10-23 13:59 UTC (permalink / raw)
To: git
Hi,
While I was debugging a crash in parsecvs while converting our CVS
repository I discovered it was because one of the CVS files had become
corrupted (truncated). This is a problem I've had before with RCS
based files which are prone to silent corruption that you won't notice
until you try and checkout an old revision of the file.
As git is fundamentally hash based it's a lot easier to determine the
health of the repository but I wonder if it's possible for silent
corruption to creep in which won't be noticed until you try and
checkout a historical commit of the tree. I notice there is a
git-verify-pack command that checks the pack files are OK. Do any of
the other commands implicitly ensure all objects in the repo are
correct and valid? git-gc?
Are there any other parts of the .git metadata that are crucial or is
it enough to say if all objects and packs match their hashes you have
all the information you may need to recover an arbitrary revision of
the repo?
--
Alex, homepage: http://www.bennee.com/~alex/
^ permalink raw reply
* Re: Verifying the whole repository
From: David Symonds @ 2008-10-23 14:05 UTC (permalink / raw)
To: Alex Bennee; +Cc: git
In-Reply-To: <b2cdc9f30810230659n15f44f64l571a0df3dbe104d9@mail.gmail.com>
On Thu, Oct 23, 2008 at 6:59 AM, Alex Bennee <kernel-hacker@bennee.com> wrote:
> As git is fundamentally hash based it's a lot easier to determine the
> health of the repository but I wonder if it's possible for silent
> corruption to creep in which won't be noticed until you try and
> checkout a historical commit of the tree. I notice there is a
> git-verify-pack command that checks the pack files are OK. Do any of
> the other commands implicitly ensure all objects in the repo are
> correct and valid? git-gc?
Try: git fsck --full --strict
Dave.
^ permalink raw reply
* Re: Verifying the whole repository
From: Alex Bennee @ 2008-10-23 14:14 UTC (permalink / raw)
To: David Symonds; +Cc: git
In-Reply-To: <ee77f5c20810230705l20339a1dj87b855bf3321f796@mail.gmail.com>
On Thu, Oct 23, 2008 at 3:05 PM, David Symonds <dsymonds@gmail.com> wrote:
> On Thu, Oct 23, 2008 at 6:59 AM, Alex Bennee <kernel-hacker@bennee.com> wrote:
>> Do any of
>> the other commands implicitly ensure all objects in the repo are
>> correct and valid? git-gc?
>
> Try: git fsck --full --strict
Ahh, I forgot that git was written by a filesystem guy ;-)
Thanks.
--
Alex, homepage: http://www.bennee.com/~alex/
^ permalink raw reply
* Re: [RFC] Zit: the git-based single file content tracker
From: Giuseppe Bilotta @ 2008-10-23 14:21 UTC (permalink / raw)
To: Nguyen Thai Ngoc Duy; +Cc: git
In-Reply-To: <fcaeb9bf0810230651j1c02de13j61238c97661c32e9@mail.gmail.com>
On Thu, Oct 23, 2008 at 3:51 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
> On 10/23/08, Giuseppe Bilotta <giuseppe.bilotta@gmail.com> wrote:
>> On Thu, Oct 23, 2008 at 2:50 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
>> > On 10/23/08, Giuseppe Bilotta <giuseppe.bilotta@gmail.com> wrote:
>> >> The principle is extremely simple: when you choose to start tracking a
>> >> file with Zit,
>> >>
>> >> zit track file
>> >>
>> >> Zit will create a directory .zit.file to hold a git repository
>> >> tracking the single file .zit.file/file, which is just a hard link to
>> >> file.
>> >
>> > Why not use one .zit repo and track each file on each own branch?.
>>
>>
>> So your proposal is to have a single .zit repo which is actually a git
>> repo and where each additional tracked file becomes its own branch,
>> and zit would take care of switching from branch to branch when zit
>> commands are called?
>
> I don't know if switching is necessary. With one file per pranch, the
> index is even not necessary.
[...]
> The history should be linear. Git (or zit) repository is just a
> container for git branches. Each branch contains only one file. Moving
> a file history is equivalent to "git push" + "git branch -D".
> Something like this (not tested):
>
> cd dst
> git init
> cd src
> git push dst local-branch:remote-branch
> git branch -D local-branch
> git gc
Looks a little too clumsy for my taste. Also, I don't like the idea of
having to enforce linear history for files, or getting rid of the
index. I would like zit to be as lightweight a wrapper for git as
possible, retaining the whole functionality.
--
Giuseppe "Oblomov" Bilotta
^ permalink raw reply
* Re: Verifying the whole repository
From: Shawn O. Pearce @ 2008-10-23 14:28 UTC (permalink / raw)
To: Alex Bennee; +Cc: git
In-Reply-To: <b2cdc9f30810230659n15f44f64l571a0df3dbe104d9@mail.gmail.com>
Alex Bennee <kernel-hacker@bennee.com> wrote:
> As git is fundamentally hash based it's a lot easier to determine the
> health of the repository but I wonder if it's possible for silent
> corruption to creep in which won't be noticed until you try and
> checkout a historical commit of the tree. I notice there is a
> git-verify-pack command that checks the pack files are OK. Do any of
> the other commands implicitly ensure all objects in the repo are
> correct and valid? git-gc?
As David pointed out, git fsck can be used to verify all of the
hashes, but git-gc also does a quick sanity check using a CRC code
when it copies data from one pack to another pack.
Unlike CVS Git has a write-once, read-many mentality, so with
the exception of git gc (err, actually the git repack it calls)
git never modifies an existing file. That really helps to reduce
the risk of corruption.
If you never do a gc or fsck operation (but still use say commit
or push into the repository) then yes, silent corruption can still
sneak up on you in the form of disk block corruption.
> Are there any other parts of the .git metadata that are crucial or is
> it enough to say if all objects and packs match their hashes you have
> all the information you may need to recover an arbitrary revision of
> the repo?
Don't forget about the loose objects under .git/objects/?? but
otherwise yes, you just need the object data. The refs under
.git/refs are also useful, but the tips can be recovered if the
refs space is lost by "git fsck --unreachable".
--
Shawn.
^ permalink raw reply
* Re: [RFC PATCH 1/5] add valgrind support in test scripts
From: Jeff King @ 2008-10-23 15:19 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: git
In-Reply-To: <alpine.DEB.1.00.0810230008430.22125@pacific.mpi-cbg.de.mpi-cbg.de>
On Thu, Oct 23, 2008 at 12:13:47AM +0200, Johannes Schindelin wrote:
> I wonder if it would not be better to scrap the t/valgrind/ directory and
> regenerate it everytime you run a test manually;
Yeah, I mentioned that in my 0/5 cover letter. The problem is where to
put it that won't impact test results, but also allow running multiple
tests simultaneously. I'm going to try sticking it in .git/valgrind in
the trash directory, which presumably won't affect any tests.
> I'd use "ln" instead of "cp"
I specifically stayed away from 'ln' for Windows portability. It looks
like for builtins, we do "ln || ln -s || cp". We can probably do the
same here.
I also failed to use git$X in the fake path, which would probably be
necessary for Windows.
> and also parse command-list.txt to catch really all of them (even if
> a dashed form is used for a builtin by mistake).
That is a little bit trickier. I don't actually want to intercept
git-am, for example, since I have no interest in running valgrind on the
shell. But it is do-able; I will give details in my response to Junio's
suggestion.
> Otherwise: good work, I like it!
Thanks.
-Peff
^ permalink raw reply
* Re: [RFC PATCH 1/5] add valgrind support in test scripts
From: Jeff King @ 2008-10-23 15:29 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Johannes Schindelin, git
In-Reply-To: <7v7i80tber.fsf@gitster.siamese.dyndns.org>
On Wed, Oct 22, 2008 at 05:14:52PM -0700, Junio C Hamano wrote:
> > I wonder if it would not be better to scrap the t/valgrind/ directory and
> > regenerate it everytime you run a test manually; I'd use "ln" instead of
> > "cp", and also parse command-list.txt to catch really all of them (even if
> > a dashed form is used for a builtin by mistake).
>
> Going one step further, I wonder if this approach can also be used to
> catch such a mistake. Install a dashed form that records the fact that it
> was called when it shouldn't, and by whom.
I think that makes sense, though it is somewhat orthogonal to valgrind.
Do we want to always set up such a fake path? It could actually be even
simpler than a wrapper; just stop adding the build directory to the
PATH, and instead have a pseudo-installation directory with the bin and
exec-path directories set up appropriately. This would more closely
model the actual installation.
I think there are actually several classes of dashed commands we need to
differentiate:
1. dashed commands which get installed in bin; these should be allowed
2. dashed commands which don't get installed in bin; these should be
flagged as an error
3. dashed commands in exec-path which are stand-alone C programs. These
should be run under valgrind.
4. dashed commands in exec-path which are scripts. These do get run,
but should not be run under valgrind.
5. dashed commands in exec-path which are builtins. These should never
get run by our test scripts, since they are "legacy" links for people
who want to put the exec-path in their PATH
Right now we always point PATH at the build directory, which has
everything. So we can't easily differentiate between '1' and '2'. I used
the $(PROGRAMS) variable in the Makefile to find '3' (as opposed to '4'
and '5').
-Peff
^ permalink raw reply
* Re: [RFC PATCH 4/5] pack-objects: avoid reading uninitalized data
From: Jeff King @ 2008-10-23 15:33 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: git
In-Reply-To: <alpine.LFD.2.00.0810222107540.26244@xanadu.home>
On Wed, Oct 22, 2008 at 09:11:16PM -0400, Nicolas Pitre wrote:
> > for (;;) {
> > - struct object_entry *entry = *list++;
> > + struct object_entry *entry;
> > struct unpacked *n = array + idx;
> > int j, max_depth, best_base = -1;
> >
> > @@ -1384,6 +1384,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,
> > progress_unlock();
> > break;
> > }
>
> ---> Please preserve the empty line here so the previous code
> chunk still appears logically separate.
>
> > + entry = *list++;
> > (*list_size)--;
> > if (!entry->preferred_base) {
> > (*processed)++;
Er, there was no empty line there (or else there would have been a '-'
line in the diff). I am happy to add it, like:
if (!*list_size) {
progress_unlock();
break;
}
entry = *list++;
(*list_size)--;
if (!entry->preferred_base)
...
if you like, but the current version seems to format it as one stanza
inside of the progress_lock/progress_unlock. Look at the version in
'master' to see what I mean.
-Peff
^ permalink raw reply
* Re: git archive
From: Nguyen Thai Ngoc Duy @ 2008-10-23 15:33 UTC (permalink / raw)
To: Deskin Miller; +Cc: kenneth johansson, git
In-Reply-To: <20081022130829.GC2015@riemann.deskinm.fdns.net>
On 10/22/08, Deskin Miller <deskinm@umich.edu> wrote:
> On Wed, Oct 22, 2008 at 08:42:01AM +0000, kenneth johansson wrote:
> > I was going to make a tar of the latest stable linux kernel.
> > Done it before but now I got a strange problem.
> >
> > >git archive --format=tar v2.6.27.2
> > fatal: Not a valid object name
>
>
> I had the same thing happen to me, while trying to make an archive of Git.
> Were you perchance working in a bare repository, as I was? I spent some time
> looking at it and I think git archive sets up the environment in the wrong
> order, though of course I never finished a patch so I'm going from memory:
>
> After looking at the code again, I think the issue is that git_config is called
> in builtin-archive.c:cmd_archive before setup_git_directory is called in
> archive.c:write_archive. The former ends up setting GIT_DIR to be '.git' even
> if you're in a bare repository. My coding skills weren't up to fixing it
> easily; moving setup_git_directory before git_config in builtin-archive caused
> last test of t5000 to fail: GIT_DIR=some/nonexistent/path git archive --list
> should still display the archive formats.
The problem affects some other commands as well. I tried the following
patch, ran "make test" and discovered "git mailinfo", "git
verify-pack", "git hash-object" and "git unpack-file". A bandage patch
is at the end of this mail. Solution is as Jeff suggested: call
setup_git_directory_gently() early.
---<---
diff --git a/environment.c b/environment.c
index 0693cd9..00ed640 100644
--- a/environment.c
+++ b/environment.c
@@ -49,14 +49,18 @@ static char *work_tree;
static const char *git_dir;
static char *git_object_dir, *git_index_file, *git_refs_dir, *git_graft_file;
+int git_dir_discovered;
static void setup_git_env(void)
{
git_dir = getenv(GIT_DIR_ENVIRONMENT);
if (!git_dir)
git_dir = read_gitfile_gently(DEFAULT_GIT_DIR_ENVIRONMENT);
- if (!git_dir)
+ if (!git_dir) {
+ if (!git_dir_discovered)
+ die("Internal error: .git must be relocated at cwd by setup_git_*");
git_dir = DEFAULT_GIT_DIR_ENVIRONMENT;
+ }
git_object_dir = getenv(DB_ENVIRONMENT);
if (!git_object_dir) {
git_object_dir = xmalloc(strlen(git_dir) + 9);
diff --git a/setup.c b/setup.c
index 78a8041..d404c21 100644
--- a/setup.c
+++ b/setup.c
@@ -368,6 +368,7 @@ const char *read_gitfile_gently(const char *path)
* We cannot decide in this function whether we are in the work tree or
* not, since the config can only be read _after_ this function was called.
*/
+extern int git_dir_discovered;
const char *setup_git_directory_gently(int *nongit_ok)
{
const char *work_tree_env = getenv(GIT_WORK_TREE_ENVIRONMENT);
@@ -472,6 +473,8 @@ const char *setup_git_directory_gently(int *nongit_ok)
}
chdir("..");
}
+ /* It is safe to call setup_git_env() now */
+ git_dir_discovered = 1;
inside_git_dir = 0;
if (!work_tree_env)
---<---
Bandage patch:
---<---
diff --git a/builtin-archive.c b/builtin-archive.c
index 432ce2a..5ea0a12 100644
--- a/builtin-archive.c
+++ b/builtin-archive.c
@@ -110,7 +110,9 @@ static const char *extract_remote_arg(int *ac,
const char **av)
int cmd_archive(int argc, const char **argv, const char *prefix)
{
const char *remote = NULL;
+ int nongit;
+ prefix = setup_git_directory_gently(&nongit);
git_config(git_default_config, NULL);
remote = extract_remote_arg(&argc, argv);
diff --git a/builtin-mailinfo.c b/builtin-mailinfo.c
index e890f7a..5d401fb 100644
--- a/builtin-mailinfo.c
+++ b/builtin-mailinfo.c
@@ -916,10 +916,9 @@ static const char mailinfo_usage[] =
int cmd_mailinfo(int argc, const char **argv, const char *prefix)
{
const char *def_charset;
+ int nongit;
- /* NEEDSWORK: might want to do the optional .git/ directory
- * discovery
- */
+ prefix = setup_git_directory_gently(&nongit);
git_config(git_default_config, NULL);
def_charset = (git_commit_encoding ? git_commit_encoding : "utf-8");
diff --git a/builtin-verify-pack.c b/builtin-verify-pack.c
index 25a29f1..35a4eb2 100644
--- a/builtin-verify-pack.c
+++ b/builtin-verify-pack.c
@@ -115,7 +115,9 @@ int cmd_verify_pack(int argc, const char **argv,
const char *prefix)
int verbose = 0;
int no_more_options = 0;
int nothing_done = 1;
+ int nongit;
+ prefix = setup_git_directory_gently(&nongit);
git_config(git_default_config, NULL);
while (1 < argc) {
if (!no_more_options && argv[1][0] == '-') {
diff --git a/hash-object.c b/hash-object.c
index 20937ff..a52b6be 100644
--- a/hash-object.c
+++ b/hash-object.c
@@ -78,19 +78,20 @@ int main(int argc, const char **argv)
const char *prefix = NULL;
int prefix_length = -1;
const char *errstr = NULL;
+ int nongit;
type = blob_type;
- git_config(git_default_config, NULL);
-
argc = parse_options(argc, argv, hash_object_options, hash_object_usage, 0);
- if (write_object) {
- prefix = setup_git_directory();
- prefix_length = prefix ? strlen(prefix) : 0;
- if (vpath && prefix)
- vpath = prefix_filename(prefix, prefix_length, vpath);
- }
+ prefix = setup_git_directory_gently(&nongit);
+ git_config(git_default_config, NULL);
+ prefix_length = prefix ? strlen(prefix) : 0;
+ if (vpath && prefix)
+ vpath = prefix_filename(prefix, prefix_length, vpath);
+
+ if (write_object && nongit)
+ die("Git repository required");
if (stdin_paths) {
if (hashstdin)
diff --git a/unpack-file.c b/unpack-file.c
index bcdc8bb..1a58d72 100644
--- a/unpack-file.c
+++ b/unpack-file.c
@@ -27,10 +27,10 @@ int main(int argc, char **argv)
if (argc != 2)
usage("git-unpack-file <sha1>");
+ setup_git_directory();
if (get_sha1(argv[1], sha1))
die("Not a valid object name %s", argv[1]);
- setup_git_directory();
git_config(git_default_config, NULL);
puts(create_temp_file(sha1));
---<---
--
Duy
^ permalink raw reply related
* Re: git performance
From: Jeff King @ 2008-10-23 16:39 UTC (permalink / raw)
To: Edward Ned Harvey; +Cc: git
In-Reply-To: <000901c93490$e0c40ed0$a24c2c70$@com>
On Wed, Oct 22, 2008 at 05:55:14PM -0400, Edward Ned Harvey wrote:
> I'm talking about 40-50,000 files, on multi-user production linux,
> which means the cache is never warm, except when I'm benchmarking.
Well, if you have a cold cache it's going to take longer. :) You should
probably benchmark if you want to know exactly how long.
> Specifically RHEL 4 with the files on NFS mount. Cold cache "svn st"
> takes ~10 mins. Warm cache 20-30 sec. Surprisingly to me,
Wow, that is awful. For comparison, "git status" from a cold on the
kernel repo takes me 17 seconds. From a warm cache, less than half a
second.
Yes, the cold cache case would probably be better with inotify, but
compared to svn, that's screaming fast. I haven't used perforce. If your
bottleneck really is stat'ing the tree, then yes, something that avoided
that might perform better (but weigh that particular optimization
against other things which might be slower).
> Out of curiosity, what are they talking about, when they say "git is
> fast?"
Well, there are the numbers above. When comparing to SVN or (god forbid)
CVS, there are order of magnitude speedups for most common operations.
> Just the fact that it's all local disk, or is there more to it
> than that? I could see - git would probably outperform perforce for
The things that generally make git fast are:
- using a compact on-disk structure (including zlib and aggressive
delta-finding) to keep your cache warm (and when it's not warm, to
get data off the disk as quickly as possible)
- the content-addressable nature of objects means we can just look at
the data we need to solve a problem. For example,
getting the history between point A and point B is "O(the number of
commits between A and B)", _not_ "O(the size of the repo)".
Viewing a log without generating diffs is "O(the number of
commits)", not "O(some combination of the number of commits and the
number of files in each commit)". Diffing two points in history is
"O(the size of the differences between the two points)" and is
totally independent of the number of commits between the two points.
- most operations are streamable. "git log >/dev/null" on the kernel
repo (about 90,000 commits) takes 8.5 seconds on my box. But it
starts generating output immediately, so it _feels_ instant, and the
rest of the data is generated while I read the first commit in my
pager.
-Peff
^ permalink raw reply
* Re: [RFC PATCH 4/5] pack-objects: avoid reading uninitalized data
From: Nicolas Pitre @ 2008-10-23 16:47 UTC (permalink / raw)
To: Jeff King; +Cc: git
In-Reply-To: <20081023153329.GC10804@coredump.intra.peff.net>
On Thu, 23 Oct 2008, Jeff King wrote:
> On Wed, Oct 22, 2008 at 09:11:16PM -0400, Nicolas Pitre wrote:
>
> > > for (;;) {
> > > - struct object_entry *entry = *list++;
> > > + struct object_entry *entry;
> > > struct unpacked *n = array + idx;
> > > int j, max_depth, best_base = -1;
> > >
> > > @@ -1384,6 +1384,7 @@ static void find_deltas(struct object_entry **list, unsigned *list_size,
> > > progress_unlock();
> > > break;
> > > }
> >
> > ---> Please preserve the empty line here so the previous code
> > chunk still appears logically separate.
> >
> > > + entry = *list++;
> > > (*list_size)--;
> > > if (!entry->preferred_base) {
> > > (*processed)++;
>
> Er, there was no empty line there (or else there would have been a '-'
> line in the diff).
Oh, right. I'm confused.
Nevermind...
Nicolas
^ permalink raw reply
* [RFC] Zit (v2): the git-based single file content tracker
From: Giuseppe Bilotta @ 2008-10-23 17:22 UTC (permalink / raw)
To: git
In-Reply-To: <gdok16$vh2$1@ger.gmane.org>
I decided to give the simpler GIT_DIR approach another go.
The reworked Zit ( git://git.oblomov.eu/zit ) works by creating
.file.git/ to track file's history. .file.git/info/excludes is
initialized to the very strong '*' pattern to ensure that things such
as git status etc only consider the actually tracked file.
The obvious advantage over the previous implementation is that we
don't rely on fragile and non-portable hardlinks. The disadvantage
is that something really bad can happen if a command fails to obey
GIT_DIR or GIT_WORK_TREE correctly.
Command delegation is made a little smarter:
zit somecommand file [args...]
gets delegated to
git somecommand [args...]
with GIT_DIR=.file.git and GIT_WORK_TREE="`pwd`", which works
surprisingly well. To prevent stupid expressions such as zit add file
file or zit commit file file, add and commit put the filename back at
the end of the parameter list.
Commands that seem to work correctly so far are init, add, log,
status, diff, remote, push, pull, and even rebase -i.
Commands that definitely need some work are rm (should it just remove
the .file.git/ dir?) and mv (hairy: we would need to rename .file.git
to .newname.git too, but rollbacks are likely to break things).
The only new command introduced by zit is zit list, which lists all
zit-tracked files in the current directory, currently in a very
braindead way (e.g. I'd like it to display the proper status, such as
C M or whatever; suggestions welcome).
On the TODO list is also some smart way to guess which file we're
talking about when no file is specified. Basically, the idea is to
check if there's only one tracked file, or only one changed tracked
file, and allow a missing file option in that case.
As usual, comments suggestions and critiques welcome.
--
Giuseppe "Oblomov" Bilotta
^ permalink raw reply
* Re: How to push to http(s) repository with authentication?
From: Josef Wolf @ 2008-10-23 17:53 UTC (permalink / raw)
To: git
In-Reply-To: <20081020182502.GH9707@raven.wolf.lan>
On Mon, Oct 20, 2008 at 08:25:02PM +0200, Josef Wolf wrote:
> I have set up a repository as described in
>
> http://www.kernel.org/pub/software/scm/git/docs/howto/setup-git-server-over-http.txt
>
> over SSL with basic authentication. DAV access works fine with konqueror,
> cadaver and and curl, using this .curlrc:
> [ ... ]
Is nobody interested in this topic? Or am I in the wrong list?
^ permalink raw reply
* Re: git archive
From: Deskin Miller @ 2008-10-23 18:21 UTC (permalink / raw)
To: Nguyen Thai Ngoc Duy; +Cc: kenneth johansson, git
In-Reply-To: <fcaeb9bf0810230833i3953a5abt2d2ba6ca1b751754@mail.gmail.com>
On Thu, Oct 23, 2008 at 10:33:31PM +0700, Nguyen Thai Ngoc Duy wrote:
> On 10/22/08, Deskin Miller <deskinm@umich.edu> wrote:
> > On Wed, Oct 22, 2008 at 08:42:01AM +0000, kenneth johansson wrote:
> > > I was going to make a tar of the latest stable linux kernel.
> > > Done it before but now I got a strange problem.
> > >
> > > >git archive --format=tar v2.6.27.2
> > > fatal: Not a valid object name
> >
> >
> > I had the same thing happen to me, while trying to make an archive of Git.
> > Were you perchance working in a bare repository, as I was? I spent some time
> > looking at it and I think git archive sets up the environment in the wrong
> > order, though of course I never finished a patch so I'm going from memory:
> >
> > After looking at the code again, I think the issue is that git_config is called
> > in builtin-archive.c:cmd_archive before setup_git_directory is called in
> > archive.c:write_archive. The former ends up setting GIT_DIR to be '.git' even
> > if you're in a bare repository. My coding skills weren't up to fixing it
> > easily; moving setup_git_directory before git_config in builtin-archive caused
> > last test of t5000 to fail: GIT_DIR=some/nonexistent/path git archive --list
> > should still display the archive formats.
>
> The problem affects some other commands as well. I tried the following
> patch, ran "make test" and discovered "git mailinfo", "git
> verify-pack", "git hash-object" and "git unpack-file". A bandage patch
> is at the end of this mail. Solution is as Jeff suggested: call
> setup_git_directory_gently() early.
Nice work. The patches look like they're on the right track (to me at least).
I'm not sure though what you want to ultimately submit as a patch; I'd suggest
both, squashed into one, since the check seems like something we'd reasonably
want no matter what.
Few comments spread around below; also, can we see some testcases for
regression? Or, does the first patch preclude the need for testcases?
Deskin Miller
> ---<---
> diff --git a/environment.c b/environment.c
> index 0693cd9..00ed640 100644
> --- a/environment.c
> +++ b/environment.c
> @@ -49,14 +49,18 @@ static char *work_tree;
>
> static const char *git_dir;
> static char *git_object_dir, *git_index_file, *git_refs_dir, *git_graft_file;
> +int git_dir_discovered;
Should this be 'int git_dir_discovered = 0;' ?
> static void setup_git_env(void)
> {
> git_dir = getenv(GIT_DIR_ENVIRONMENT);
> if (!git_dir)
> git_dir = read_gitfile_gently(DEFAULT_GIT_DIR_ENVIRONMENT);
> - if (!git_dir)
> + if (!git_dir) {
> + if (!git_dir_discovered)
> + die("Internal error: .git must be relocated at cwd by setup_git_*");
> git_dir = DEFAULT_GIT_DIR_ENVIRONMENT;
> + }
> git_object_dir = getenv(DB_ENVIRONMENT);
> if (!git_object_dir) {
> git_object_dir = xmalloc(strlen(git_dir) + 9);
> diff --git a/setup.c b/setup.c
> index 78a8041..d404c21 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -368,6 +368,7 @@ const char *read_gitfile_gently(const char *path)
> * We cannot decide in this function whether we are in the work tree or
> * not, since the config can only be read _after_ this function was called.
> */
> +extern int git_dir_discovered;
> const char *setup_git_directory_gently(int *nongit_ok)
> {
> const char *work_tree_env = getenv(GIT_WORK_TREE_ENVIRONMENT);
> @@ -472,6 +473,8 @@ const char *setup_git_directory_gently(int *nongit_ok)
> }
> chdir("..");
> }
> + /* It is safe to call setup_git_env() now */
> + git_dir_discovered = 1;
>
> inside_git_dir = 0;
> if (!work_tree_env)
> ---<---
>
>
> Bandage patch:
>
> ---<---
> diff --git a/builtin-archive.c b/builtin-archive.c
> index 432ce2a..5ea0a12 100644
> --- a/builtin-archive.c
> +++ b/builtin-archive.c
> @@ -110,7 +110,9 @@ static const char *extract_remote_arg(int *ac,
> const char **av)
> int cmd_archive(int argc, const char **argv, const char *prefix)
> {
> const char *remote = NULL;
> + int nongit;
>
> + prefix = setup_git_directory_gently(&nongit);
Here and elsewhere, the 'nongit' variable isn't used.
setup_git_directory_gently can be passed a NULL pointer, why not do that?
> git_config(git_default_config, NULL);
>
> remote = extract_remote_arg(&argc, argv);
> diff --git a/builtin-mailinfo.c b/builtin-mailinfo.c
> index e890f7a..5d401fb 100644
> --- a/builtin-mailinfo.c
> +++ b/builtin-mailinfo.c
> @@ -916,10 +916,9 @@ static const char mailinfo_usage[] =
> int cmd_mailinfo(int argc, const char **argv, const char *prefix)
> {
> const char *def_charset;
> + int nongit;
>
> - /* NEEDSWORK: might want to do the optional .git/ directory
> - * discovery
> - */
> + prefix = setup_git_directory_gently(&nongit);
Same 'nongit' issue.
> git_config(git_default_config, NULL);
>
> def_charset = (git_commit_encoding ? git_commit_encoding : "utf-8");
> diff --git a/builtin-verify-pack.c b/builtin-verify-pack.c
> index 25a29f1..35a4eb2 100644
> --- a/builtin-verify-pack.c
> +++ b/builtin-verify-pack.c
> @@ -115,7 +115,9 @@ int cmd_verify_pack(int argc, const char **argv,
> const char *prefix)
> int verbose = 0;
> int no_more_options = 0;
> int nothing_done = 1;
> + int nongit;
>
> + prefix = setup_git_directory_gently(&nongit);
Same 'nongit' issue.
> git_config(git_default_config, NULL);
> while (1 < argc) {
> if (!no_more_options && argv[1][0] == '-') {
> diff --git a/hash-object.c b/hash-object.c
> index 20937ff..a52b6be 100644
> --- a/hash-object.c
> +++ b/hash-object.c
> @@ -78,19 +78,20 @@ int main(int argc, const char **argv)
> const char *prefix = NULL;
> int prefix_length = -1;
> const char *errstr = NULL;
> + int nongit;
>
> type = blob_type;
>
> - git_config(git_default_config, NULL);
> -
> argc = parse_options(argc, argv, hash_object_options, hash_object_usage, 0);
>
> - if (write_object) {
> - prefix = setup_git_directory();
> - prefix_length = prefix ? strlen(prefix) : 0;
> - if (vpath && prefix)
> - vpath = prefix_filename(prefix, prefix_length, vpath);
> - }
> + prefix = setup_git_directory_gently(&nongit);
> + git_config(git_default_config, NULL);
> + prefix_length = prefix ? strlen(prefix) : 0;
> + if (vpath && prefix)
> + vpath = prefix_filename(prefix, prefix_length, vpath);
> +
> + if (write_object && nongit)
> + die("Git repository required");
I'd move this check up to just after setup_git_directory_gently.
> if (stdin_paths) {
> if (hashstdin)
> diff --git a/unpack-file.c b/unpack-file.c
> index bcdc8bb..1a58d72 100644
> --- a/unpack-file.c
> +++ b/unpack-file.c
> @@ -27,10 +27,10 @@ int main(int argc, char **argv)
>
> if (argc != 2)
> usage("git-unpack-file <sha1>");
> + setup_git_directory();
> if (get_sha1(argv[1], sha1))
> die("Not a valid object name %s", argv[1]);
>
> - setup_git_directory();
> git_config(git_default_config, NULL);
>
> puts(create_temp_file(sha1));
> ---<---
> --
> Duy
^ permalink raw reply
* RE: git performance
From: Daniel Barkalow @ 2008-10-23 18:31 UTC (permalink / raw)
To: Edward Ned Harvey; +Cc: git
In-Reply-To: <000901c93490$e0c40ed0$a24c2c70$@com>
On Wed, 22 Oct 2008, Edward Ned Harvey wrote:
> Out of curiosity, what are they talking about, when they say "git is
> fast?" Just the fact that it's all local disk, or is there more to it
> than that? I could see - git would probably outperform perforce for
> versioning of large files (let's say iso files) to benefit from
> sustained local disk IO, while perforce would probably outperform
> anything I can think of, operating on thousands of tiny files, because
> it will never walk the tree.
It shouldn't be too hard to make git work like perforce with respect to
walking the tree. git keeps an index of the stat() info it saw when it
last looked at files, and only looks at the contents of files whose stat()
info has changed. In order to have it work like perforce, it would just
need to have a flag in the stat() info index for "don't even bother",
which it would use for files that aren't "open"; for files with this flag,
the check for index freshness would always say it's fresh without looking
at the filesystem. Then you'd just have a config option to check out files
as "not open" (and not writeable), and have a "git open" program that
would chmod files and get their real stat info.
Of course, git is tuned for cases where the modify/build/test cycle
requires stat() (or worse) on every file.
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: [PATCH] Only update the cygwin-related configuration during state auto-setup
From: Alex Riesen @ 2008-10-23 18:36 UTC (permalink / raw)
To: Mark Levedahl; +Cc: gitster, spearce, dpotapov, git
In-Reply-To: <81b0412b0810230607sfea05ddm62bd03f837fc922e@mail.gmail.com>
Err... It should be: "Only update the cygwin-related configuration
during stat auto-setup".
^ permalink raw reply
* Re: How to push to http(s) repository with authentication?
From: Alex Riesen @ 2008-10-23 19:05 UTC (permalink / raw)
To: Josef Wolf; +Cc: git
In-Reply-To: <20081023175336.GA3526@raven.wolf.lan>
Josef Wolf, Thu, Oct 23, 2008 19:53:37 +0200:
> On Mon, Oct 20, 2008 at 08:25:02PM +0200, Josef Wolf wrote:
>
> > I have set up a repository as described in
> >
> > http://www.kernel.org/pub/software/scm/git/docs/howto/setup-git-server-over-http.txt
> >
> > over SSL with basic authentication. DAV access works fine with konqueror,
> > cadaver and and curl, using this .curlrc:
> > [ ... ]
>
> Is nobody interested in this topic? Or am I in the wrong list?
--cacert <CA certificate>
(SSL) Tells curl to use the specified certificate file to verify
the peer. The file may contain multiple CA certificates. The
certificate(s) must be in PEM format.
curl recognizes the environment variable named ’CURL_CA_BUNDLE’
if that is set, and uses the given path as a path to a CA cert
bundle. This option overrides that variable.
The windows version of curl will automatically look for a CA
certs file named ´curl-ca-bundle.crt´, either in the same direc‐
tory as curl.exe, or in the Current Working Directory, or in any
folder along your PATH.
If curl is built against the NSS SSL library then this option
tells curl the nickname of the CA certificate to use within the
NSS database defined by the environment variable SSL_DIR (or by
default /etc/pki/nssdb). If the NSS PEM PKCS#11 module (lib‐
nsspem.so) is available then PEM files may be loaded.
If this option is used several times, the last one will be used.
Does this (setting $CURL_CA_BUNDLE) help?
^ permalink raw reply
* [PATCH] index-pack: don't leak leaf delta result
From: Nicolas Pitre @ 2008-10-23 19:05 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
Another (but minor this time) fallout from commit 9441b61dc5.
Signed-off-by: Nicolas Pitre <nico@cam.org>
diff --git a/index-pack.c b/index-pack.c
index 7f97faa..fe75332 100644
--- a/index-pack.c
+++ b/index-pack.c
@@ -550,8 +550,10 @@ static void find_unresolved_deltas(struct base_data *base,
find_delta_children(&base_spec, &ofs_first, &ofs_last);
}
- if (ref_last == -1 && ofs_last == -1)
+ if (ref_last == -1 && ofs_last == -1) {
+ free(base->data);
return;
+ }
link_base_data(prev_base, base);
Nicolas
^ permalink raw reply related
* [PATCH] git-svn: change dashed git-config to git config
From: Deskin Miller @ 2008-10-23 19:21 UTC (permalink / raw)
To: git; +Cc: normalperson
---
When testing git-svn changes, I'll often use git-svn.perl directly on a
repository of mine, to see how things work. Recently I had this happen:
$ git-svn.perl info
Use of uninitialized value in scalar chomp at ../git/git-svn.perl line 1202.
Use of uninitialized value in concatenation (.) or string at ../git/git-svn.perl line 1203.
but the expected output showed up correctly, giving the SVN-like info.
git svn info appeared to work just fine though, despite there being essentially
no difference between the two. However, if I do the following, this is what I
see:
$ env /usr/libexec/git-core/git-svn info
Can't exec "git-config": No such file or directory at /usr/libexec/git-core/git-svn line 1202.
Use of uninitialized value in scalar chomp at /usr/libexec/git-core/git-svn line 1202.
Again, the SVN info would still appear.
Regardless, I think we want to change this call to use the non-dashed form.
Deskin Miller
git-svn.perl | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/git-svn.perl b/git-svn.perl
index 33e1b50..2e68c68 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -1126,7 +1126,7 @@ sub read_repo_config {
my $v = $opts->{$o};
my ($key) = ($o =~ /^([a-zA-Z\-]+)/);
$key =~ s/-//g;
- my $arg = 'git-config';
+ my $arg = 'git config';
$arg .= ' --int' if ($o =~ /[:=]i$/);
$arg .= ' --bool' if ($o !~ /[:=][sfi]$/);
if (ref $v eq 'ARRAY') {
--
1.6.0.2.554.g3041b
^ permalink raw reply related
* [RFC PATCH v2] fetch-pack: log(n)-transmission find_common()
From: Thomas Rast @ 2008-10-23 19:38 UTC (permalink / raw)
To: git
In-Reply-To: <1222217303-29792-1-git-send-email-trast@student.ethz.ch>
Replaces the existing simple history search with a more sophisticated
algorithm:
1) Walk history with exponentially increasing stride lengths; i.e.,
send the 1st commit, then the 2nd after that, then the 4th after
that, and so on.
2) Bisect the resulting intervals.
Combined with tracking the "outstanding haves" so that we can detect
which sha1s were never ACKed by upload-pack (and hence not common),
this gives O(log(n)) required "have" lines.
Unfortunately this cannot work if the server sends fake ACKs, so we
introduce a capability 'exp-stride' which instructs upload-pack to
disable ok_to_give_up(). (Which incidentally saves the server a lot
of work.)
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---
This is a simple resend of v2, in the hope of attracting some
discussion or at least encouraging words this time around.
Here are the timings I took when I wrote the last mail:
-- 8< --
For the timings below, I ran the (new) daemon locally, with two repos:
- a 'clone --mirror' of my own git.git, with a branch 'next' 525
commits from the above cc185a6
- a 'clone --mirror' of the completely unrelated adps.git, which
shares no objects with git.git
Some not really statistically sound timings, usually best of 3-5:
## A stripped down git.git, with only one branch and no tags
$ git branch -a
* master
$ git tag
$ git log -1 --pretty=oneline master
cc185a6a8ac24737a26ec4b40cc401c2db8b2e97 Start draft release notes for 1.6.0.3
## I still have the old (dashed) forms in /usr/bin from RPMs
$ /usr/bin/git --version
git version 1.5.6
## (1a) "Updating" from the git.git that is further ahead (OLD)
$ time /usr/bin/git-fetch-pack -k git://localhost/git next
[...]
real 0m1.004s
user 0m0.228s
sys 0m0.028s
## (1b) "Updating" from the git.git that is further ahead (NEW)
$ time git fetch-pack -k git://localhost/git next
[...]
real 0m0.977s
user 0m0.208s
sys 0m0.068s
## (2a) Fetching the unrelated repo from scratch (OLD)
$ time /usr/bin/git-fetch-pack -k git://localhost/adps master
[...]
real 0m9.560s
user 0m0.720s
sys 0m0.128s
## (2b) Fetching the unrelated repo from scratch (NEW)
$ time git fetch-pack -k git://localhost/adps master
[...]
real 0m0.596s
user 0m0.372s
sys 0m0.008s
## (3a) Fetching over the network from real repo (OLD)
$ time /usr/bin/git-fetch-pack git://git.kernel.org/pub/scm/git/git.git next
[...]
user 0m0.528s
sys 0m0.136s
## (3b) Fetching over the network from real repo (NEW)
$ time git fetch-pack git://git.kernel.org/pub/scm/git/git.git next
[...]
user 0m0.540s
sys 0m0.180s
## Add more refs to make it more interesting
$ git remote add origin ~/dev/git
$ git fetch --tags origin
## (4a) like 1a, but with lots of tags
$ time /usr/bin/git-fetch-pack -k git://localhost/git next
[...]
real 0m1.075s
user 0m0.452s
sys 0m0.048s
## (4b) like 1b, but with lots of tags
$ time git fetch-pack -k git://localhost/git next
[...]
real 0m0.837s
user 0m0.236s
sys 0m0.040s
Clearly, this approach does solve the issue I mentioned in the
motivation earlier in the thread, where the initial fetch of
completely disjoint repositories takes _ages_. Somewhat to my own
surprise, it seems to do quite well in _all_ cases even though it
aggressively digs through history.
Note, however, that the timings aren't really solid; the final write
of the pack, which I assume is fsync()ed, seems to have a big impact
on the execution time. Test (3) obviously depends more on network
performance than anything else; and it uses the old (linear)
algorithm, but in the new implementation.
-- >8 --
builtin-fetch-pack.c | 477 +++++++++++++++++++++++++++++++++++++++++---------
upload-pack.c | 7 +-
2 files changed, 400 insertions(+), 84 deletions(-)
diff --git a/builtin-fetch-pack.c b/builtin-fetch-pack.c
index 372bfa2..9ab84ab 100644
--- a/builtin-fetch-pack.c
+++ b/builtin-fetch-pack.c
@@ -25,6 +25,12 @@ static const char fetch_pack_usage[] =
#define COMMON_REF (1U << 2)
#define SEEN (1U << 3)
#define POPPED (1U << 4)
+#define BISECTED_FW (1U << 5)
+#define BISECTED_BW (1U << 6)
+#define NOTCOMMON (1U << 7)
+#define HAS_INFO (1U << 8)
+#define CLEAN_MARKS (COMMON | COMMON_REF | SEEN | POPPED | BISECTED_FW \
+ | BISECTED_BW | NOTCOMMON)
static int marked;
@@ -34,133 +40,412 @@ static int marked;
*/
#define MAX_IN_VAIN 256
-static struct commit_list *rev_list;
-static int non_common_revs, multi_ack, use_sideband;
+struct work_list
+{
+ struct work_list *next;
+ struct commit *commit;
+ unsigned int stride; /* current stride length between commits */
+ unsigned int steps; /* steps left to go before emitting again */
+ unsigned int depth; /* sideline depth */
+ unsigned int bisect;
+};
+
+/* "recursion stack" of our strategy of choosing sha1s to transmit */
+static struct work_list *work_list = NULL;
-static void rev_list_push(struct commit *commit, int mark)
+/* list of commits not ACKed yet */
+static struct commit_list *outstanding = NULL;
+static struct commit_list *outstanding_end = NULL;
+
+struct commit_info
{
- if (!(commit->object.flags & mark)) {
- commit->object.flags |= mark;
+ struct commit_list *children;
+};
- if (!(commit->object.parsed))
- if (parse_commit(commit))
- return;
- insert_by_date(commit, &rev_list);
+static int non_common_revs = 1, multi_ack, use_sideband, exp_stride_algorithm;
- if (!(commit->object.flags & COMMON))
- non_common_revs++;
+static struct commit_info *get_commit_info(struct commit *commit)
+{
+ if (commit->object.flags & HAS_INFO)
+ return commit->util;
+
+ struct commit_info *info = xmalloc(sizeof(struct commit_info));
+ info->children = NULL;
+ commit->util = info;
+ commit->object.flags |= HAS_INFO;
+ return info;
+}
+
+static void work_list_insert(struct work_list *work_item)
+{
+ struct work_list **pp = &work_list;
+ struct work_list *p;
+
+ while ((p = *pp)) {
+ if (p->bisect || work_item->bisect) {
+ if (p->bisect > work_item->bisect)
+ break;
+ } else {
+ if (p->depth > work_item->depth)
+ break;
+ else if (p->commit->date < work_item->commit->date)
+ break;
+ }
+
+ pp = &p->next;
}
+
+ work_item->next = p;
+ *pp = work_item;
}
-static int rev_list_insert_ref(const char *path, const unsigned char *sha1, int flag, void *cb_data)
+static void add_child(struct commit *parent, struct commit *child)
+{
+ struct commit_info *info = get_commit_info(parent);
+ struct commit_list *item;
+ for (item = info->children; item; item = item->next)
+ if (item->item == child)
+ break;
+ if (!item)
+ commit_list_insert(child, &(info->children));
+}
+
+
+/*
+ Marks all (known) children of this commit NOTCOMMON. We call this
+ for all sha1s the server did not ACK.
+*/
+
+static void mark_not_common(struct commit *commit)
+{
+ struct commit_info *info = get_commit_info(commit);
+ struct commit_list *child;
+
+ assert(!(commit->object.flags & COMMON));
+
+ if (commit->object.flags & NOTCOMMON)
+ return;
+
+ commit->object.flags |= NOTCOMMON;
+
+ for (child = info->children; child; child = child->next)
+ mark_not_common(child->item);
+}
+
+
+static void check_parsed_and_mark(struct commit *commit)
+{
+ struct commit_list *parent;
+
+ if (!commit && commit->object.parsed)
+ return;
+
+ parse_commit(commit);
+
+ for (parent = commit->parents; parent; parent = parent->next) {
+ add_child(parent->item, commit);
+ if (parent->item->object.flags & NOTCOMMON)
+ mark_not_common(commit);
+ }
+}
+
+
+static void rev_list_push(struct commit *commit, unsigned int stride,
+ unsigned int steps, unsigned int depth)
+{
+ struct work_list *work_item;
+
+ check_parsed_and_mark(commit);
+
+ work_item = xmalloc(sizeof(struct work_list));
+ work_item->commit = commit;
+ work_item->stride = stride;
+ work_item->steps = steps;
+ work_item->depth = depth;
+ work_item->bisect = 0;
+ work_list_insert(work_item);
+}
+
+static int rev_list_insert_ref(const char *path, const unsigned char *sha1,
+ int flag, void *cb_data)
{
struct object *o = deref_tag(parse_object(sha1), path, 0);
- if (o && o->type == OBJ_COMMIT)
- rev_list_push((struct commit *)o, SEEN);
+ if (o && o->type == OBJ_COMMIT && !(o->flags & SEEN))
+ rev_list_push((struct commit *)o, 1, 1, 0);
return 0;
}
-static int clear_marks(const char *path, const unsigned char *sha1, int flag, void *cb_data)
+static int clear_marks(const char *path, const unsigned char *sha1, int flag,
+ void *cb_data)
{
struct object *o = deref_tag(parse_object(sha1), path, 0);
if (o && o->type == OBJ_COMMIT)
- clear_commit_marks((struct commit *)o,
- COMMON | COMMON_REF | SEEN | POPPED);
+ clear_commit_marks((struct commit *)o, CLEAN_MARKS);
return 0;
}
+
/*
- This function marks a rev and its ancestors as common.
- In some cases, it is desirable to mark only the ancestors (for example
- when only the server does not yet know that they are common).
+ Mark commits backwards through history as COMMON.
*/
-static void mark_common(struct commit *commit,
- int ancestors_only, int dont_parse)
+static void mark_common(struct commit *commit, int ancestors_only, int dont_parse)
{
- if (commit != NULL && !(commit->object.flags & COMMON)) {
- struct object *o = (struct object *)commit;
+ struct commit_list *parents;
+ struct object *o;
- if (!ancestors_only)
- o->flags |= COMMON;
+ if (commit == NULL || commit->object.flags & (COMMON_REF|COMMON))
+ return;
- if (!(o->flags & SEEN))
- rev_list_push(commit, SEEN);
- else {
- struct commit_list *parents;
-
- if (!ancestors_only && !(o->flags & POPPED))
- non_common_revs--;
- if (!o->parsed && !dont_parse)
- if (parse_commit(commit))
- return;
-
- for (parents = commit->parents;
- parents;
- parents = parents->next)
- mark_common(parents->item, 0, dont_parse);
- }
+ if (!dont_parse && !(commit->object.parsed))
+ check_parsed_and_mark(commit);
+
+ o = (struct object *)commit;
+
+ if (!ancestors_only)
+ o->flags |= COMMON;
+
+ for (parents = commit->parents; parents; parents = parents->next) {
+ add_child(parents->item, commit);
+ mark_common(parents->item, 0, dont_parse);
}
}
+
+
/*
- Get the next rev to send, ignoring the common.
+ Queue all revs in the list for simple backward search. Does not
+ requeue already SEEN commits.
*/
-static const unsigned char* get_rev(void)
+static void queue_list(struct commit *child, struct commit_list *list,
+ unsigned int stride, unsigned int steps,
+ unsigned int depth)
{
- struct commit *commit = NULL;
+ while (list) {
+ add_child(list->item, child);
+ if (!(list->item->object.flags & SEEN))
+ rev_list_push(list->item, stride, steps, depth);
+ list = list->next;
+ }
+}
+
+
+/*
+ Queues a commit for bisection
+*/
+
+static void setup_bisect(struct commit *commit, int depth)
+{
+ struct work_list *work_item = xmalloc(sizeof(struct work_list));
+
+ work_item->commit = commit;
+ work_item->bisect = 1+depth;
+ work_item->depth = 0;
+ work_list_insert(work_item);
+}
+
+
+/*
+ Queues a list of commits for bisection
+*/
+
+static void setup_bisect_all (struct commit_list* list, int depth)
+{
+ while (list) {
+ setup_bisect(list->item, depth);
+ list = list->next;
+ }
+}
+
+
+/*
+ Backwards search. If we use exp_stride_algorithm, it takes
+ exponential strides between commits chosen.
+*/
+
+static struct commit* get_rev_stride(struct work_list *work_item)
+{
+ struct commit *commit = work_item->commit;
+ unsigned int steps = work_item->steps;
+
+ while (steps++ < work_item->stride-1 && commit->parents
+ && !(commit->object.flags & (SEEN|COMMON|COMMON_REF))) {
+ /* all but the first parent line are queued for later */
+ if (commit->parents)
+ queue_list(commit, commit->parents->next,
+ work_item->stride, steps,
+ work_item->depth+1);
+
+ /* follow the first parent line directly */
+ commit->object.flags |= SEEN;
+ add_child(commit->parents->item, commit);
+ commit = commit->parents->item;
+ check_parsed_and_mark(commit);
+ }
+
+ if (commit->object.flags & (SEEN|COMMON)) {
+ /* already been here! */
+ commit = NULL;
+ } else if (commit->object.flags & COMMON_REF) {
+ /* this came from a ref; we stop here, but need to
+ * emit it so the server knows too */
+ commit->object.flags |= POPPED|SEEN|BISECTED_BW;
+ if (exp_stride_algorithm)
+ setup_bisect(commit, 0);
+ } else {
+ /* usual case: this is news to us, so we scan further
+ * back. note that the first-depth bisection only
+ * needs to go forward */
+ commit->object.flags |= POPPED|SEEN|BISECTED_BW;
+ queue_list(commit, commit->parents,
+ exp_stride_algorithm ? work_item->stride*2 : 1,
+ 0, work_item->depth);
+ if (exp_stride_algorithm)
+ setup_bisect(commit, 0);
+ }
+
+ free(work_item);
+
+ return commit;
+}
- while (commit == NULL) {
- unsigned int mark;
- struct commit_list *parents;
- if (rev_list == NULL || non_common_revs == 0)
+/*
+ Bisection state of the exponential stride algorithm: scans forward
+ (backward) for a non-common/already-sent (common/already-sent,
+ resp.) commit, then emits the middle of that range, and re-queues.
+
+ This means we bisect the discovered range in log(n) transmissions,
+ although we do make O(n) steps through history.
+*/
+
+static struct commit* get_rev_bisect(struct work_list *work_item)
+{
+ struct commit *full;
+ struct commit *half;
+ int half_step;
+ struct commit_info *info;
+ unsigned int flags = work_item->commit->object.flags;
+
+ if (!(flags & (COMMON|BISECTED_BW))) {
+ /* We don't know anything about the history _backward_
+ * from this, so search it */
+
+ full = work_item->commit;
+ full->object.flags |= BISECTED_BW;
+ half = work_item->commit;
+ half_step = 0;
+
+ while (full && full->parents) {
+ full = full->parents->item;
+ half_step ^= 1;
+ if (half_step)
+ half = half->parents->item;
+ if (full->object.flags & (POPPED|COMMON))
+ break;
+ }
+
+ /* also insert the same bisection again so we can try
+ * forward too */
+ work_list_insert(work_item);
+
+ if (full->object.flags & POPPED
+ && !(full->object.flags & NOTCOMMON)
+ && !(half->object.flags & (COMMON|POPPED))) {
+ setup_bisect(half, work_item->bisect);
+ half->object.flags |= POPPED;
+ return half;
+ } else {
return NULL;
+ }
+ }
- commit = rev_list->item;
- if (!commit->object.parsed)
- parse_commit(commit);
- parents = commit->parents;
-
- commit->object.flags |= POPPED;
- if (!(commit->object.flags & COMMON))
- non_common_revs--;
-
- if (commit->object.flags & COMMON) {
- /* do not send "have", and ignore ancestors */
- commit = NULL;
- mark = COMMON | SEEN;
- } else if (commit->object.flags & COMMON_REF)
- /* send "have", and ignore ancestors */
- mark = COMMON | SEEN;
- else
- /* send "have", also for its ancestors */
- mark = SEEN;
-
- while (parents) {
- if (!(parents->item->object.flags & SEEN))
- rev_list_push(parents->item, mark);
- if (mark & COMMON)
- mark_common(parents->item, 1, 0);
- parents = parents->next;
+ if (!(flags & (NOTCOMMON|BISECTED_FW))) {
+ /* We don't know anything about the history _forward_
+ * from this, so search it */
+
+ full = work_item->commit;
+ full->object.flags |= BISECTED_FW;
+ half = work_item->commit;
+ half_step = 0;
+
+ while (full && (info=get_commit_info(full))->children) {
+ setup_bisect_all(info->children->next, work_item->bisect);
+ full = info->children->item;
+ half_step ^= 1;
+ if (half_step) {
+ info = get_commit_info(half);
+ half = info->children->item;
+ }
+ if (full->object.flags & (POPPED|NOTCOMMON))
+ break;
}
- rev_list = rev_list->next;
+ if (full->object.flags & POPPED
+ && !(full->object.flags & COMMON)
+ && !(half->object.flags & (NOTCOMMON|POPPED))) {
+ setup_bisect(half, work_item->bisect);
+ half->object.flags |= POPPED;
+ free(work_item);
+ return half;
+ }
}
- return commit->object.sha1;
+ free(work_item);
+ return NULL;
}
+
+/*
+ Get the next revision to send.
+*/
+
+static struct commit *get_rev(void)
+{
+ struct commit *commit = NULL;
+
+ while (commit == NULL && work_list) {
+ struct work_list *work_item = NULL;
+
+ work_item = work_list;
+ /* we update the pointer early so the get_rev
+ * functions can free() or reuse the work_item as
+ * required */
+ work_list = work_item->next;
+
+ commit = work_item->commit;
+
+ check_parsed_and_mark(commit);
+
+ if (work_item->bisect) {
+ commit = get_rev_bisect(work_item);
+ } else {
+ commit = get_rev_stride(work_item);
+ }
+ }
+
+ return commit;
+}
+
+static void pop_outstanding()
+{
+ struct commit_list *item = outstanding;
+ outstanding = item->next;
+ free(item);
+}
+
+
static int find_common(int fd[2], unsigned char *result_sha1,
struct ref *refs)
{
int fetching;
int count = 0, flushes = 0, retval;
+ struct commit *commit;
const unsigned char *sha1;
unsigned in_vain = 0;
int got_continue = 0;
@@ -192,7 +477,7 @@ static int find_common(int fd[2], unsigned char *result_sha1,
}
if (!fetching)
- packet_write(fd[1], "want %s%s%s%s%s%s%s%s\n",
+ packet_write(fd[1], "want %s%s%s%s%s%s%s%s%s\n",
sha1_to_hex(remote),
(multi_ack ? " multi_ack" : ""),
(use_sideband == 2 ? " side-band-64k" : ""),
@@ -200,6 +485,7 @@ static int find_common(int fd[2], unsigned char *result_sha1,
(args.use_thin_pack ? " thin-pack" : ""),
(args.no_progress ? " no-progress" : ""),
(args.include_tag ? " include-tag" : ""),
+ (exp_stride_algorithm ? " exp-stride" : ""),
" ofs-delta");
else
packet_write(fd[1], "want %s\n", sha1_to_hex(remote));
@@ -243,13 +529,25 @@ static int find_common(int fd[2], unsigned char *result_sha1,
flushes = 0;
retval = -1;
- while ((sha1 = get_rev())) {
+ while ((commit = get_rev())) {
+ sha1 = commit->object.sha1;
packet_write(fd[1], "have %s\n", sha1_to_hex(sha1));
if (args.verbose)
fprintf(stderr, "have %s\n", sha1_to_hex(sha1));
in_vain++;
+
+ if (outstanding) {
+ commit_list_insert(commit, &(outstanding_end->next));
+ outstanding_end = outstanding_end->next;
+ } else {
+ commit_list_insert(commit, &outstanding);
+ outstanding_end = outstanding;
+ }
+
if (!(31 & ++count)) {
int ack;
+ int unwound = 0;
+ struct commit_list *item;
packet_flush(fd[1]);
flushes++;
@@ -274,12 +572,23 @@ static int find_common(int fd[2], unsigned char *result_sha1,
} else if (ack == 2) {
struct commit *commit =
lookup_commit(result_sha1);
+ while (commit != outstanding->item) {
+ mark_not_common(outstanding->item);
+ pop_outstanding();
+ unwound++;
+ }
+ pop_outstanding();
+ unwound++;
mark_common(commit, 0, 1);
retval = 0;
in_vain = 0;
got_continue = 1;
}
} while (ack);
+ while (unwound++ < 32) {
+ mark_not_common(outstanding->item);
+ pop_outstanding();
+ }
flushes--;
if (got_continue && MAX_IN_VAIN < in_vain) {
if (args.verbose)
@@ -445,9 +754,8 @@ static int everything_local(struct ref **refs, int nr_match, char **match)
continue;
if (!(o->flags & SEEN)) {
- rev_list_push((struct commit *)o, COMMON_REF | SEEN);
-
- mark_common((struct commit *)o, 1, 1);
+ rev_list_push((struct commit *)o, 1, 0, 0);
+ o->flags |= COMMON_REF;
}
}
@@ -597,6 +905,11 @@ static struct ref *do_fetch_pack(int fd[2],
fprintf(stderr, "Server supports side-band\n");
use_sideband = 1;
}
+ if (server_supports("exp-stride")) {
+ if (args.verbose)
+ fprintf(stderr, "Server supports exp-stride\n");
+ exp_stride_algorithm = 1;
+ }
if (everything_local(&ref, nr_match, match)) {
packet_flush(fd[1]);
goto all_done;
diff --git a/upload-pack.c b/upload-pack.c
index e5adbc0..14012ee 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -37,6 +37,7 @@ static unsigned int timeout;
*/
static int use_sideband;
static int debug_fd;
+static int exp_stride_algorithm;
static void reset_timeout(void)
{
@@ -414,7 +415,7 @@ static int get_common_commits(void)
if (!prefixcmp(line, "have ")) {
switch (got_sha1(line+5, sha1)) {
case -1: /* they have what we do not */
- if (multi_ack && ok_to_give_up())
+ if (multi_ack && !exp_stride_algorithm && ok_to_give_up())
packet_write(1, "ACK %s continue\n",
sha1_to_hex(sha1));
break;
@@ -501,6 +502,8 @@ static void receive_needs(void)
no_progress = 1;
if (strstr(line+45, "include-tag"))
use_include_tag = 1;
+ if (strstr(line+45, "exp-stride"))
+ exp_stride_algorithm = 1;
/* We have sent all our refs already, and the other end
* should have chosen out of them; otherwise they are
@@ -573,7 +576,7 @@ static int send_ref(const char *refname, const unsigned char *sha1, int flag, vo
{
static const char *capabilities = "multi_ack thin-pack side-band"
" side-band-64k ofs-delta shallow no-progress"
- " include-tag";
+ " include-tag exp-stride";
struct object *o = parse_object(sha1);
if (!o)
--
tg: (759ad19..) t/fetch-pack-speedup (depends on: origin/master)
^ permalink raw reply related
* clean/smudge filters for pdf files
From: Leo Razoumov @ 2008-10-23 19:44 UTC (permalink / raw)
To: git
I am trying to improve storage efficiency for PDF files in a git repo.
Following earlier discussions in this list I am trying to set up
proper clean/smudge filters. What follows is my current setup
# in ~/.gitconfig
[filter "pdf"]
clean = "pdftk - output - uncompress"
smudge = "pdftk - output - compress"
# in .gitattributes
*.pdf filter=pdf
Unfortunately, it seems as though that pdftk uncompress followed by
pdftk compress do not leave the file invariant. I tried several
uncompress+compress iterations and the file still keep changing (the
size though stays the same).
Is there any other alternative way to store PDF files in git repo more
efficiently?
Any alternative to pdftk on Linux?
--Leo--
^ permalink raw reply
* Re: [JGIT PATCH 8/8] Define a basic merge API, and a two-way tree merge strategy
From: Robin Rosenberg @ 2008-10-23 21:14 UTC (permalink / raw)
To: Shawn O. Pearce; +Cc: git
In-Reply-To: <1223932217-4771-9-git-send-email-spearce@spearce.org>
Hi, Shawn
Shouldn't testTrivialTwoWay_disjointhistories() work?
The two trees have nothing in common and so should be trivially mergeable.
-- robin
>From cef2695431e368da616a1e9c8de3e5e419854a4c Mon Sep 17 00:00:00 2001
From: Robin Rosenberg <robin.rosenberg@dewire.com>
Date: Thu, 23 Oct 2008 23:09:10 +0200
Subject: [EGIT PATCH] Simple merge test
---
.../org/spearce/jgit/merge/SimpleMergeTest.java | 28 ++++++++++++++++++++
1 files changed, 28 insertions(+), 0 deletions(-)
create mode 100644 org.spearce.jgit.test/tst/org/spearce/jgit/merge/SimpleMergeTest.java
diff --git a/org.spearce.jgit.test/tst/org/spearce/jgit/merge/SimpleMergeTest.java b/org.spearce.jgit.test/tst/org/spearce/jgit/merge/SimpleMergeTest.java
new file mode 100644
index 0000000..8ec1c7f
--- /dev/null
+++ b/org.spearce.jgit.test/tst/org/spearce/jgit/merge/SimpleMergeTest.java
@@ -0,0 +1,28 @@
+package org.spearce.jgit.merge;
+
+import java.io.IOException;
+
+import org.spearce.jgit.lib.ObjectId;
+import org.spearce.jgit.lib.RepositoryTestCase;
+
+public class SimpleMergeTest extends RepositoryTestCase {
+
+ public void testTrivialTwoWay_noway() throws IOException {
+ Merger ourMerger = MergeStrategy.SIMPLE_TWO_WAY_IN_CORE.newMerger(db);
+ boolean merge = ourMerger.merge(new ObjectId[] { db.resolve("a"), db.resolve("c") });
+ assertFalse(merge);
+ }
+
+ public void testTrivialTwoWay_disjointhistories() throws IOException {
+ Merger ourMerger = MergeStrategy.SIMPLE_TWO_WAY_IN_CORE.newMerger(db);
+ boolean merge = ourMerger.merge(new ObjectId[] { db.resolve("a"), db.resolve("c~4") });
+ assertTrue(merge);
+ }
+
+ public void testTrivialTwoWay_ok() throws IOException {
+ Merger ourMerger = MergeStrategy.SIMPLE_TWO_WAY_IN_CORE.newMerger(db);
+ boolean merge = ourMerger.merge(new ObjectId[] { db.resolve("a^0^0^0"), db.resolve("a^0^0^1") });
+ assertTrue(merge);
+ assertEquals(db.mapTree("a^0^0").getId(), ourMerger.getResultTreeId());
+ }
+}
--
1.6.0.2.308.gef4a
^ permalink raw reply related
* Re: clean/smudge filters for pdf files
From: Pierre Habouzit @ 2008-10-23 21:32 UTC (permalink / raw)
To: Leo Razoumov; +Cc: git
In-Reply-To: <ee2a733e0810231244u1510cdc8y7b27af9b8fdda1e0@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 1182 bytes --]
On Thu, Oct 23, 2008 at 07:44:39PM +0000, Leo Razoumov wrote:
> I am trying to improve storage efficiency for PDF files in a git repo.
> Following earlier discussions in this list I am trying to set up
> proper clean/smudge filters. What follows is my current setup
>
> # in ~/.gitconfig
> [filter "pdf"]
> clean = "pdftk - output - uncompress"
> smudge = "pdftk - output - compress"
>
> # in .gitattributes
> *.pdf filter=pdf
>
> Unfortunately, it seems as though that pdftk uncompress followed by
> pdftk compress do not leave the file invariant. I tried several
> uncompress+compress iterations and the file still keep changing (the
> size though stays the same).
> Is there any other alternative way to store PDF files in git repo more
> efficiently?
> Any alternative to pdftk on Linux?
actually it uses some kind of zlib algorithm so that's pretty normal you
don't have the same result with a packer. Maybe one could write a tool
like pristine-tar for that purpose.
--
·O· Pierre Habouzit
··O madcoder@debian.org
OOO http://www.madism.org
[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply
* Conference in near
From: Anthony Murphy @ 2008-10-23 22:04 UTC (permalink / raw)
To: git
The 2008 Hard Money National Conference in near
All the information is at:
http://hardmoneyparty.com
or call us today at 858-736-7788 for the event details.
to be taken of our list please:
http://hardmoneyparty.com/uns/ or, send us a letter to PBMSII, 5580 la jolla blvd #153 La Jolla, Ca 92037
--------------------------------------------
.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox