* Finally implement "git log --follow"
From: Linus Torvalds @ 2007-06-19 21:22 UTC (permalink / raw)
To: Junio C Hamano, Git Mailing List
Ok, I've really held off doing this too damn long, because I'm lazy, and I
was always hoping that somebody else would do it.
But no, people keep asking for it, but nobody actually did anything, so I
decided I might as well bite the bullet, and instead of telling people
they could add a "--follow" flag to "git log" to do what they want to do,
I decided that it looks like I just have to do it for them..
The code wasn't actually that complicated, in that the diffstat for this
patch literally says "70 insertions(+), 1 deletions(-)", but I will have
to admit that in order to get to this fairly simple patch, you did have to
know and understand the internal git diff generation machinery pretty
well, and had to really be able to follow how commit generation interacts
with generating patches and generating the log.
So I suspect that while I was right that it wasn't that hard, I might have
been expecting too much of random people - this patch does seem to be
firmly in the core "Linus or Junio" territory.
To make a long story short: I'm sorry for it taking so long until I just
did it.
I'm not going to guarantee that this works for everybody, but you really
can just look at the patch, and after the appropriate appreciative noises
("Ooh, aah") over how clever I am, you can then just notice that the code
itself isn't really that complicated.
All the real new code is in the new "try_to_follow_renames()" function. It
really isn't rocket science: we notice that the pathname we were looking
at went away, so we start a full tree diff and try to see if we can
instead make that pathname be a rename or a copy from some other previous
pathname. And if we can, we just continue, except we show *that*
particular diff, and ever after we use the _previous_ pathname.
One thing to look out for: the "rename detection" is considered to be a
singular event in the _linear_ "git log" output! That's what people want
to do, but I just wanted to point out that this patch is *not* carrying
around a "commit,pathname" kind of pair and it's *not* going to be able to
notice the file coming from multiple *different* files in earlier history.
IOW, if you use "git log --follow", then you get the stupid CVS/SVN kind
of "files have single identities" kind of semantics, and git log will just
pick the identity based on the normal move/copy heuristics _as_if_ the
history could be linearized.
Put another way: I think the model is broken, but given the broken model,
I think this patch does just about as well as you can do. If you have
merges with the same "file" having different filenames over the two
branches, git will just end up picking _one_ of the pathnames at the point
where the newer one goes away. It never looks at multiple pathnames in
parallel.
And if you understood all that, you probably didn't need it explained, and
if you didn't understand the above blathering, it doesn't really mtter to
you. What matters to you is that you can now do
git log -p --follow builtin-rev-list.c
and it will find the point where the old "rev-list.c" got renamed to
"builtin-rev-list.c" and show it as such.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
----
NOTE! For obvious enough reasons, I limited the rename following to just
*one* pathname. If somebody wants to extend it to any more, be my guest.
Also, I really might have some silly bug somewhere.
"It Works For Me(tm)", and I tested it with both the git archive and the
kernel (block/ll_rw_block.c) and it worked beautifully, but I'll also
admit that the patch is a bit _too_ clever for my taste normally. The
patch actually looks more straightforward than it really is: that "hook
directly into diff_tree_sha1()" thing is just too damn clever for words.
People who want to improve on it should get rid of the memory leak I
introduced - I decided to not bother cleaning up the whole rename diff
queue, I just reset it. I'm lazy, and I'm a *man*. I do the rough manly
stuff, others can clean up after me.
*Burp*. That hit the spot. *Scratch*
Linus
---
builtin-log.c | 5 ++++
diff.c | 2 +
diff.h | 1 +
revision.c | 4 ++-
tree-diff.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
5 files changed, 70 insertions(+), 1 deletions(-)
diff --git a/builtin-log.c b/builtin-log.c
index 0aede76..a26a010 100644
--- a/builtin-log.c
+++ b/builtin-log.c
@@ -58,6 +58,11 @@ static void cmd_log_init(int argc, const char **argv, const char *prefix,
argc = setup_revisions(argc, argv, rev, "HEAD");
if (rev->diffopt.pickaxe || rev->diffopt.filter)
rev->always_show_header = 0;
+ if (rev->diffopt.follow_renames) {
+ rev->always_show_header = 0;
+ if (rev->diffopt.nr_paths != 1)
+ usage("git logs can only follow renames on one pathname at a time");
+ }
for (i = 1; i < argc; i++) {
const char *arg = argv[i];
if (!strcmp(arg, "--decorate")) {
diff --git a/diff.c b/diff.c
index 4aa9bbc..9938969 100644
--- a/diff.c
+++ b/diff.c
@@ -2210,6 +2210,8 @@ int diff_opt_parse(struct diff_options *options, const char **av, int ac)
}
else if (!strcmp(arg, "--find-copies-harder"))
options->find_copies_harder = 1;
+ else if (!strcmp(arg, "--follow"))
+ options->follow_renames = 1;
else if (!strcmp(arg, "--abbrev"))
options->abbrev = DEFAULT_ABBREV;
else if (!prefixcmp(arg, "--abbrev=")) {
diff --git a/diff.h b/diff.h
index a7ee6d8..9fd6d44 100644
--- a/diff.h
+++ b/diff.h
@@ -55,6 +55,7 @@ struct diff_options {
full_index:1,
silent_on_remove:1,
find_copies_harder:1,
+ follow_renames:1,
color_diff:1,
color_diff_words:1,
has_changes:1,
diff --git a/revision.c b/revision.c
index 1f4590b..7834bb1 100644
--- a/revision.c
+++ b/revision.c
@@ -1230,7 +1230,9 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, const ch
if (revs->prune_data) {
diff_tree_setup_paths(revs->prune_data, &revs->pruning);
- revs->prune_fn = try_to_simplify_commit;
+ /* Can't prune commits with rename following: the paths change.. */
+ if (!revs->diffopt.follow_renames)
+ revs->prune_fn = try_to_simplify_commit;
if (!revs->full_diff)
diff_tree_setup_paths(revs->prune_data, &revs->diffopt);
}
diff --git a/tree-diff.c b/tree-diff.c
index 852498e..42924e9 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -3,6 +3,7 @@
*/
#include "cache.h"
#include "diff.h"
+#include "diffcore.h"
#include "tree.h"
static char *malloc_base(const char *base, int baselen, const char *path, int pathlen)
@@ -290,6 +291,59 @@ int diff_tree(struct tree_desc *t1, struct tree_desc *t2, const char *base, stru
return 0;
}
+/*
+ * Does it look like the resulting diff might be due to a rename?
+ * - single entry
+ * - not a valid previous file
+ */
+static inline int diff_might_be_rename(void)
+{
+ return diff_queued_diff.nr == 1 &&
+ !DIFF_FILE_VALID(diff_queued_diff.queue[0]->one);
+}
+
+static void try_to_follow_renames(struct tree_desc *t1, struct tree_desc *t2, const char *base, struct diff_options *opt)
+{
+ struct diff_options diff_opts;
+ const char *paths[2];
+ int i;
+
+ diff_setup(&diff_opts);
+ diff_opts.recursive = 1;
+ diff_opts.detect_rename = DIFF_DETECT_RENAME;
+ diff_opts.output_format = DIFF_FORMAT_NO_OUTPUT;
+ diff_opts.single_follow = opt->paths[0];
+ paths[0] = NULL;
+ diff_tree_setup_paths(paths, &diff_opts);
+ if (diff_setup_done(&diff_opts) < 0)
+ die("unable to set up diff options to follow renames");
+ diff_tree(t1, t2, base, &diff_opts);
+ diffcore_std(&diff_opts);
+
+ /* NOTE! Ignore the first diff! That was the old one! */
+ for (i = 1; i < diff_queued_diff.nr; i++) {
+ struct diff_filepair *p = diff_queued_diff.queue[i];
+
+ /*
+ * Found a source? Not only do we use that for the new
+ * diff_queued_diff, we also use that as the path in
+ * the future!
+ */
+ if ((p->status == 'R' || p->status == 'C') && !strcmp(p->two->path, opt->paths[0])) {
+ diff_queued_diff.queue[0] = p;
+ opt->paths[0] = xstrdup(p->one->path);
+ diff_tree_setup_paths(opt->paths, opt);
+ break;
+ }
+ }
+
+ /*
+ * Then, ignore any but the first entry! It might be the old one,
+ * or it might be the rename/copy we found
+ */
+ diff_queued_diff.nr = 1;
+}
+
int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const char *base, struct diff_options *opt)
{
void *tree1, *tree2;
@@ -306,6 +360,11 @@ int diff_tree_sha1(const unsigned char *old, const unsigned char *new, const cha
init_tree_desc(&t1, tree1, size1);
init_tree_desc(&t2, tree2, size2);
retval = diff_tree(&t1, &t2, base, opt);
+ if (opt->follow_renames && diff_might_be_rename()) {
+ init_tree_desc(&t1, tree1, size1);
+ init_tree_desc(&t2, tree2, size2);
+ try_to_follow_renames(&t1, &t2, base, opt);
+ }
free(tree1);
free(tree2);
return retval;
^ permalink raw reply related
* Re: Finally implement "git log --follow"
From: Linus Torvalds @ 2007-06-19 21:35 UTC (permalink / raw)
To: Junio C Hamano, Git Mailing List
In-Reply-To: <alpine.LFD.0.98.0706191358180.3593@woody.linux-foundation.org>
On Tue, 19 Jun 2007, Linus Torvalds wrote:
>
> People who want to improve on it should get rid of the memory leak I
> introduced - I decided to not bother cleaning up the whole rename diff
> queue, I just reset it. I'm lazy, and I'm a *man*. I do the rough manly
> stuff, others can clean up after me.
>
> *Burp*. That hit the spot. *Scratch*
On that note, if people want to decide that "git log" should *default* to
follow when one filename is given, that's ok by me. A few warnings:
- You should only do it for the "single file" case, and that's very
*different* from the "single pattern" case. I often do "git log" on a
single directory, and that had better not start doing any "filename
following" by default.
So before turning on "follow" automatically, it should be checked that
that exact file exists right now (might be as simplistic as just doing
a "lstat()" and verifying that it's a file or symlink in the current
working tree)
- the commit simplification isn't done. So you cannot do "--follow"
together with somethign that wants a contiguous history (like gitk) and
uses "--parents".
This is pretty fundamental. The commit simplification is a separate
phase over the (non-linear!) commit space, and the "git log --follow"
logic is really a *different* logic over the _linear_ space (namely the
streaming "log output" space).
Two totally different things, in other words. And not compatible.
So if we want to turn on "follow" by default, then I'm certainly ok with
it, but it needs to be done with care. so I'd almost suggest just adding
an alias instead, aka
git config alias.follow 'log --follow'
and then doing
git follow filename
rather than changing the behaviour of "git log" itself.
Linus
^ permalink raw reply
* Re: StGIT rebasing safeguard
From: Catalin Marinas @ 2007-06-19 22:12 UTC (permalink / raw)
To: Yann Dirson; +Cc: git
In-Reply-To: <20070614213032.GR6992@nan92-1-81-57-214-146.fbx.proxad.net>
On 14/06/07, Yann Dirson <ydirson@altern.org> wrote:
> When the parent branch is a rewinding one (eg. an stgit stack), then
> the old version of the patch will be turned to unreachable by
> pull/rebase, and we probably have even no way of telling stgit that it
> is indeed expected, since the parent stack is a local one. My own
> workflow on StGIT is affected by the issue, since my "bugs" stack is
> forked off my "master" stack (but hopefully an hydra will help me ;).
If I understand correctly, is this the case where you do a 'stg
commit'? This command is meant for branches that are never rebased
(i.e. my master stgit branch). For this branch one wouldn't have a
remote branch configured and hence git fetch shouldn't do anything.
> That makes me suspecting the reachability approach is a dead-end, and
> we should either get back to the approach of recording old-base, or
> find another solution.
Maybe defaulting to 'git pull' and only use 'git fetch' if explicitly
asked. A git-pull would still keep the old patch accessible from HEAD.
--
Catalin
^ permalink raw reply
* Re: StGIT rebasing safeguard
From: Catalin Marinas @ 2007-06-19 22:18 UTC (permalink / raw)
To: Yann Dirson; +Cc: git
In-Reply-To: <b0943d9e0706191512l5f19b80v5b5be9151be13025@mail.gmail.com>
On 19/06/07, Catalin Marinas <catalin.marinas@gmail.com> wrote:
> On 14/06/07, Yann Dirson <ydirson@altern.org> wrote:
> > When the parent branch is a rewinding one (eg. an stgit stack), then
> > the old version of the patch will be turned to unreachable by
> > pull/rebase, and we probably have even no way of telling stgit that it
> > is indeed expected, since the parent stack is a local one. My own
> > workflow on StGIT is affected by the issue, since my "bugs" stack is
> > forked off my "master" stack (but hopefully an hydra will help me ;).
>
> If I understand correctly, is this the case where you do a 'stg
> commit'? This command is meant for branches that are never rebased
> (i.e. my master stgit branch). For this branch one wouldn't have a
> remote branch configured and hence git fetch shouldn't do anything.
I got confused - you were talking about 'stg rebase' rather than the
'pull' strategy. But the question remains - are you referring to the
user running 'stg commit' and losing this commit after a rebase?
The rebase should be equivalent to a git-reset but with
popping/pushing of the patches one the stack. Once committed, the
patch is no longer managed by StGIT and therefore we shouldn't care.
--
Catalin
^ permalink raw reply
* Re: [StGIT PATCH 2/4] Abstract a PatchSet object out of Series.
From: Catalin Marinas @ 2007-06-19 22:32 UTC (permalink / raw)
To: Yann Dirson; +Cc: git
In-Reply-To: <20070615201439.GT6992@nan92-1-81-57-214-146.fbx.proxad.net>
On 15/06/07, Yann Dirson <ydirson@altern.org> wrote:
> Indeed one of the things that naturally come to mind after hydra, is
> to abstract a parent class above PatchSet and Patch, and allow those
> to be mostly used everywhere one of them is allowed.
>
> That way we can have a (sub-)stack anywhere between 2 patches in a
> stack, and that should I think address the need you describe. But
> that would also allow to have an hydra built of single patches instead
> of stacks, which would be quite similar to how darcs organizes
> patches. Combinations are endless, and I don't even count the
> possibility of adding new structures besides stacks and hydras :)
If you can track the patch dependencies, it would be enough to no
longer have a stack but a pool of patches (like darcs but without
exact patch commuting, the diff3 merging provides a good approximation
of this operation and in a much faster time). Simple patch dependency
could be detected by trying to apply a patch without others (it can be
optimised so that only patches touching common files should be
checked).
> > It's not a bug. The import command just uses the e-mail sender or a
> > "From:" line before the patch description (see the default mail
> > template). It doesn't check the sign lines (it is following the kernel
> > patch submission guidelines).
>
> We could surely improve things (and I'm not suggesting we should look
> at sign lines). Eg, by having stg-mail add an Author pseudo-header
> when the patch author is different from the sender, and having
> stg-import use that when available.
But the 'stg mail' command already adds a "From:" line in the body
(with the default template) and 'import' checks for 'From:' or
'Author:' which override the e-mail sender. You might be using a
different e-mail template.
--
Catalin
^ permalink raw reply
* Re: StGIT rebasing safeguard
From: Yann Dirson @ 2007-06-19 22:37 UTC (permalink / raw)
To: Catalin Marinas; +Cc: git
In-Reply-To: <b0943d9e0706191518o6396a2c5u1fb4f0953dc85aa@mail.gmail.com>
On Tue, Jun 19, 2007 at 11:18:56PM +0100, Catalin Marinas wrote:
> On 19/06/07, Catalin Marinas <catalin.marinas@gmail.com> wrote:
> >On 14/06/07, Yann Dirson <ydirson@altern.org> wrote:
> >> When the parent branch is a rewinding one (eg. an stgit stack), then
> >> the old version of the patch will be turned to unreachable by
> >> pull/rebase, and we probably have even no way of telling stgit that it
> >> is indeed expected, since the parent stack is a local one. My own
> >> workflow on StGIT is affected by the issue, since my "bugs" stack is
> >> forked off my "master" stack (but hopefully an hydra will help me ;).
> >
> >If I understand correctly, is this the case where you do a 'stg
> >commit'? This command is meant for branches that are never rebased
> >(i.e. my master stgit branch). For this branch one wouldn't have a
> >remote branch configured and hence git fetch shouldn't do anything.
>
> I got confused - you were talking about 'stg rebase' rather than the
> 'pull' strategy. But the question remains - are you referring to the
> user running 'stg commit' and losing this commit after a rebase?
Not at all. I'm talking about an stgit stack, whose base is the head
of another stack (eg. a "pu" branch). When you want to rebase, the
old heads/pu is only reachable from your stack base.
Maybe we can use the parent's reflog, but I'm not sure it would cover
all cases, and a reflog can possibly be expired before the information
in there gets used.
> The rebase should be equivalent to a git-reset but with
> popping/pushing of the patches one the stack. Once committed, the
> patch is no longer managed by StGIT and therefore we shouldn't care.
That is another issue - and I rather believe we should not allow a
user to do that without --force, if what he committed will be
unreachable after rebasing. If he intended to loose it, he would
usually rather have used "stg delete" than such a convoluted sequence
of actions :)
Best regards,
--
Yann.
^ permalink raw reply
* Re: [StGIT PATCH 0/9] Refactoring of command handling
From: Yann Dirson @ 2007-06-19 22:41 UTC (permalink / raw)
To: Catalin Marinas; +Cc: git
In-Reply-To: <20070616213615.14941.31187.stgit@gandelf.nowhere.earth>
On Sun, Jun 17, 2007 at 12:00:28AM +0200, Yann Dirson wrote:
> Refactor command definition with a Command class.
That patch (3/9, and the main patch in the series) was most probably
blocked from the list because of its size. The whole series is anyway
available at http://repo.or.cz/w/stgit/ydirson.git (together with even
more unreleased patches - beware ;)
More generally, I'll try to --sign those patches in my stack that I
consider stable and have been (or are in the queue fo being) sent to
Catalin.
Best regards,
--
Yann
^ permalink raw reply
* Re: how to move with history?
From: Jakub Narebski @ 2007-06-19 22:59 UTC (permalink / raw)
To: git
In-Reply-To: <20070619192834.GS3037@cs-wsok.swansea.ac.uk>
Oliver Kullmann wrote:
> (b) More convenient, the upcoming "git-filter-branch" apparently
> makes filtering out easier.
> (c) Or, apparently more powerful, with "cg-admin-rewritehis"
> (part of the cogito-tool) we have quite a powerful tool
> for creating a branch with a (re-created and modified)
> history (the documentation explicitely mentions how to
> remove a file from history --- that is, in the new branch).
git-filter-branch _is_ cg-admin-rewritehist (or at least should be).
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* [PATCH]: tree-walk.h: Warning fix
From: Luiz Fernando N. Capitulino @ 2007-06-20 2:11 UTC (permalink / raw)
To: gitster; +Cc: git
Code that is using libgit.a, and hence including GIT's headers, may
get this warning:
"""
../tree-walk.h: In function 'tree_entry_len':
../tree-walk.h:25: warning: cast discards qualifiers from pointer target type
../tree-walk.h:25: warning: cast discards qualifiers from pointer target type
"""
This happens because the cast used in tree_entry_len() is discarding
the const qualifier (thanks to Shawn Pearce by noticing this).
Signed-off-by: Luiz Fernando N. Capitulino <lcapitulino@gmail.com>
diff --git a/tree-walk.h b/tree-walk.h
index ee747ab..625198f 100644
--- a/tree-walk.h
+++ b/tree-walk.h
@@ -22,7 +22,7 @@ static inline const unsigned char *tree_entry_extract(struct tree_desc *desc, co
static inline int tree_entry_len(const char *name, const unsigned char *sha1)
{
- return (char *)sha1 - (char *)name - 1;
+ return (const char *)sha1 - (const char *)name - 1;
}
void update_tree_entry(struct tree_desc *);
^ permalink raw reply related
* Re: Stupid quoting...
From: Johannes Schindelin @ 2007-06-20 2:19 UTC (permalink / raw)
To: David Kastrup; +Cc: git
In-Reply-To: <86645kutow.fsf@lola.quinscape.zz>
Hi,
[sorry for responding so late, your mail got stuck in the GWB-like spam
filter.]
On Tue, 19 Jun 2007, David Kastrup wrote:
> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>
> > Hi,
> >
> > On Tue, 19 Jun 2007, David Kastrup wrote:
> >
> >> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> >>
> >> > Don't just throw away backwards compatibility, only because it does
> >> > not fit your wishes.
> >>
> >> There is no backwards compatibility involved here _at_ _all_.
> >
> > I was not talking about Git here. The specification for SMTP is not
> > going to change just because you want it. There are still mail
> > servers out there which speak 7-bit, and the standard requires you
> > to cope with them.
>
> Is there a reason you elide all the relevant material before replying?
> I repeat: this is the task of MIME, uuencode or a number of other
> mechanisms.
The problem there, of course, is that you still might want to reply to the
patch, even if the name was chosen as non-ASCII (which is a sin, if you
believe in UNIX).
Usually, comments are not done on the filenames, so they can be as escaped
as they want in an email, as long as the commenter still recognizes their
names.
> git is not a mail transport system, and there are far too many other
> problems in unarmored mail (like spaces, wrapping and other stuff) that
> it would make any sense to mangle diffs and other material in a manner
> that makes it quite unprocessable for _both_ human readers as well as
> scripts intended to process them.
There you have a point. If the name is non-ASCII, it uses a specific
encoding. if the human reader has a different encoding set in her display,
is it any better to display garbled characters (possibly leaving the
console in a corrupted state), or to display escaped characters?
And scripts have been known to get encodings all wrong, so I think the
escaping is the best way out, absent a perfect knowledge of what encoding
the file name was meant for.
> Anyway, it has become quite clear from this exchange that you have
> already made the decision not to be convinced by me and will not be
> deterred from that, even though the problem is not the one you initially
> tried deriding me for (spaces in filenames).
I am sorry. No, really, I am sorry that you received it as derision. By
all means, it was _not_ meant as that. The problem was on my side, not
yours: I simply did not get that you were talking about non-ASCII
characters, even if you were talking about them.
> Hopefully some developer with less of an attitude towards non-ASCII
> usage will find himself able to follow the arguments with some more
> objectivity.
>
> I don't see our discourse leading anywhere: the points have been made.
I would really, really, really like to see a solution. Alas, I cannot
think of one, other than _forcing_ the developers to use ASCII-only
filenames.
Note that there is no convention yet in Git to state which encoding your
filenames are supposed to use. And in fact, we already had a fine example
in git.git why this is particularly difficult. MacOSX is too clever to be
true, in that it gladly takes filenames in one encoding, but reads those
filenames out in _another_ encoding. Thus, a "git add <filename>" can well
end up in git-status saying that a file was deleted, and another file
(actually the same, but in a different encoding) is untracked.
Again, I would be _so_ glad if you solved the problem, now that I actually
understand it.
Ciao,
Dscho
^ permalink raw reply
* Debugging strange "corrupt pack" errors on SuSE 9
From: Martin Langhoff @ 2007-06-20 2:36 UTC (permalink / raw)
To: Git Mailing List, jonathan.newman
A colleage at work is using git to manage code updates to a heavily
firewalled machine at a client site. In a nutshell, we "push" the
interesting code to a repo on usb-stick, and we pull from it on the
client's machine.
We do some coding and testing on that machine, so ocassionally we
bring some patches back.
Now the working repo on the client machine has started to die with
"corrupt pack" errors. I am trying to get my hands on the literal
error messages, and exact software versions installed. Right now all I
know is that it is SuSE 9 x86, git 1.4.x, cogito .17.x . The error
appears on git-diff, git-fsck-objects --full
We did bring a copy of the working copy with the "corrupt pack" to our
office, and here git (v1.5.1 and 1.5.2) thinks it's perfectly well.
So I am a bit puzzled - while we try to get 1.5.x on the client
machine and see what happens, is there anything that could be causing
this? Any additional tests that we should run?
cheers,
martin
^ permalink raw reply
* Re: Versioning file system
From: Kyle Moffett @ 2007-06-20 2:43 UTC (permalink / raw)
To: Bron Gondwana
Cc: Bryan Henderson, Jack Stone, Andrew Morton, alan, H. Peter Anvin,
linux-fsdevel, LKML Kernel, Al Viro, git
In-Reply-To: <20070619075857.GA2944@brong.net>
On Jun 19, 2007, at 03:58:57, Bron Gondwana wrote:
> On Mon, Jun 18, 2007 at 11:10:42PM -0400, Kyle Moffett wrote:
>> On Jun 18, 2007, at 13:56:05, Bryan Henderson wrote:
>>>> The question remains is where to implement versioning: directly
>>>> in individual filesystems or in the vfs code so all filesystems
>>>> can use it?
>>>
>>> Or not in the kernel at all. I've been doing versioning of the
>>> types I described for years with user space code and I don't
>>> remember feeling that I compromised in order not to involve the
>>> kernel.
>>
>> What I think would be particularly interesting in this domain is
>> something similar in concept to GIT, except in a file-system:
>
> [...snip...]
>
> It can work, but there's one big pain at the file level: no mmap.
IMHO it's actually not that bad. The "gitfs" would divide larger
files up into manageable chunks (say 4MB) which could be quickly
SHA-1ed. When a file is mmapped and partially modified, the SHA-1
would be marked as locally invalid, but since mmap() loses most
consistency guarantees that's OK. A time or writeout based "commit"
scheme might still freeze, SHA-1, and write-out the page at regular
intervals without the program's knowledge, but since you only have to
SHA-1 the relatively-small 4MB chunk (which is about to hit disk
anyways), it's not a significant time penalty. Even if under memory
pressure and swapping data out to disk you don't have to update the
SHA-1 and create a new commit as long as you keep a reference to the
object stored in the volume header somewhere and maintain the "SHA-1
out-of-date" bit.
A program which carefully uses msync() would be fine, of course (with
proper configuration) as that would create a new commit as appropriate.
Since mmap() is poorly defined on network filesystems in the absence
of msync(), I don't see that such behaviour would be a problem. And
it certainly would be fine on local filesystems as there you can just
stuff the "SHA-1 out-of-date" bit and a reference to the parent
commit and path in the object itself. Then you just need to keep a
useful reference to that object in a table somewhere in the volume
and you're set.
> If you don't want to support mmap it can work reasonably happily,
> though you may want to keep your sha1 (or other digest) state as
> well as the final digest so you can cheaply calculate the digest
> for a small append without walking the entire file. You may also
> want to keep state checkpoints every so often along a big file so
> that truncates don't cost too much to recalculate.
That may be worth it even if the file is divided into 4MB chunks (or
other configurable value), but it would need benchmarking.
> Luckily in a userspace VFS that's only accessed via FTP and DAV we
> can support a limited set of operations (basically create, append,
> read, delete) You don't get that luxury for a general purpose
> filesystem, and that's the problem. There will always be
> particular usage patterns (especially something that mmaps or seeks
> and touches all over the place like a loopback mounted filesystem
> or a database file) that just dodn't work for file-level sha1s.
I'd think that loopback-mounted filesystems wouldn't be that difficult
1) Set the SHA-1 block size appropriately to divide the big file
into a bunch of little manageable files. Could conceivably be multi-
layered like directories, depending on the size of the file.
2) Mark the file as exempt from normal commits (IE: without
special syscalls or fsync/msync() on the file itself, it is never
updated in the tree objects.
3) Set up the loopback device to call the gitfs commit code when
it receives barriers or flushes from the parent filesystem.
And database files aren't a big issue. I have yet to see a networked
filesystem which you could stick a MySQL database on it from one node
and expect to get useful/recent read results from other nodes. If
you really wanted something like that for such a "gitfs", you could
just add code to MySQL to create a gitfs commit every N transactions
and not otherwise. The best part is: that would make online MySQL
backups from another node trivial! Just pick any arbitrary
appropriate commit object and mount that object, then "cp -a
mysql_db_dir mysql_backup_dir". That's not to say it wouldn't have a
performance penalty, but for some people the performance penalty
might be worth it.
Oh, and for those programs which want multi-master replication, this
makes it ten times easier:
1) Put each master-server on a different gitfs branch
2) Write your program as gitfs aware. Make it create gitfs
commits at appropriate times (so the data is accessible from other
nodes).
3) Come up with a useful non-interactive database-file merge
algorithm. Useful examples of different kinds of merge engines may
be found in the git project. This should take $BASE_VERSION,
$NEWVERSION1, $NEWVERSION2, and produce a $MERGEDVERSION. A good
algorithm should probably pick a safe default and save a "conflict"
entry in the face of conflicting changes.
4) Hook your merge algorithm into the gitfs mechanics using some
to-be-defined API.
5) Whenever your software does a database-file commit it sends
out a little notification to the other nodes (maybe using a gitfs API?)
6) Run a periodic (as defined by the admin yet again) thread on
each node which does branch merging. When two or more branches have
different SHA-1 sums the servers will rotate the merging task between
them. The thus-selected server will merge changes from the other
server(s) into its current working copy. With 2 servers this means
that the maximum delay between one server making a change and the
other server seeing it will be 2 times the merge interval.
7) For small pools of servers a simple rotated-merge-master
algorithm would work. For larger pools you would need to come up
with some logarithmic rotating-merge-node algorithm to evenly divide
the work of propagating changes across all nodes.
> It does have some lovely properties though. I'd enjoy working in
> an envionment that didn't look much like POSIX but had the strong
> guarantees and auditability that addressing by sha1 buys you.
I'd like to think we can have our cake and eat it too :-D. POSIX
requirements should be doable on the local system and can be mimiced
well enough on networked filesystems (albeit with update latency)
that most programs won't care. If you're the only person modifying
files on gitfs, regardless of what node they are stored on, it should
have the same behavior as local files (since with gitfs caching they
would *become* local files too :-D). The few programs that do care
about POSIX atomicity across networked filesystems (which is already
mostly implementation defined) could probably be updated to map gitfs
commits and merges into their own internal transactions and do just
fine.
Cheers,
Kyle Moffett
^ permalink raw reply
* Errors install git-1.5.2.2 on 64-bit AIX
From: Dongsheng Song @ 2007-06-20 2:45 UTC (permalink / raw)
Cc: Junio C Hamano, git
After I modified Makefile:
NO_OPENSSL=1
NO_CURL=1
NO_EXPAT=1
CFLAGS = -maix64 -g -O2 -Wall
LDFLAGS = -b64 -lz
AR = ar -X64
Make succeed:
bash-3.00# gnumake
GIT_VERSION = 1.5.2.2
* new build flags or prefix
CC convert-objects.o
...
SUBDIR perl
cp private-Error.pm blib/lib/Error.pm
cp Git.pm blib/lib/Git.pm
Manifying blib/man3/private-Error.3
Manifying blib/man3/Git.3
SUBDIR templates
gcc -maix64 -g -O2 -Wall -DNO_OPENSSL
-DSHA1_HEADER='"mozilla-sha1/sha1.h"'
-DETC_GITCONFIG='"/usr/git/etc/gitconfig"' -DNO_STRCASESTR
-DNO_STRLCPY -o test-chmtime -lz test-chmtime.c
gcc -maix64 -g -O2 -Wall -DNO_OPENSSL
-DSHA1_HEADER='"mozilla-sha1/sha1.h"'
-DETC_GITCONFIG='"/usr/git/etc/gitconfig"' -DNO_STRCASESTR
-DNO_STRLCPY -o test-genrandom -lz test-genrandom.c
But install failed:
bash-3.00# gnumake install
SUBDIR git-gui
SUBDIR perl
SUBDIR templates
install -d -m755 '/usr/git/bin'
getopt: illegal option -- d
getopt: illegal option -- 7
getopt: illegal option -- 5
getopt: illegal option -- 5
Usage: install [-c dira] [-f dirb] [-i] [-m] [-M mode] [-O owner]
[-G group] [-S] [-n dirc] [-o] [-s] file [dirx ...]
gnumake: *** [install] Error 2
Thanks for some help.
---
Dongsheng
^ permalink raw reply
* Re: Errors install git-1.5.2.2 on 64-bit AIX
From: Linus Torvalds @ 2007-06-20 3:21 UTC (permalink / raw)
To: Dongsheng Song; +Cc: Junio C Hamano, Git Mailing List
In-Reply-To: <4b3406f0706191945j1a489743qfacdcafd7f7d73d4@mail.gmail.com>
On Wed, 20 Jun 2007, Dongsheng Song wrote:
>
> But install failed:
> [ ... ]
> Usage: install [-c dira] [-f dirb] [-i] [-m] [-M mode] [-O owner]
> [-G group] [-S] [-n dirc] [-o] [-s] file [dirx ...]
> gnumake: *** [install] Error 2
Do you possibly have a "ginstall" somewhere in addition to the GNU make?
If so, just make the "INSTALL" macro in the Makefile point to that instead
of the (apparently totally broken) regular "install" program on AIX.
Maybe it's called "gnuinstall".
That said, the installation is really just a matter of copying, so you
*could* just replace the uses of "install" with either "-mkdir" or "cp"
depending on whether it's used to make sure a directory exists, or to
actually copy the programs.
Linus
^ permalink raw reply
* Re: Debugging strange "corrupt pack" errors on SuSE 9
From: Nicolas Pitre @ 2007-06-20 3:33 UTC (permalink / raw)
To: Martin Langhoff; +Cc: Git Mailing List, jonathan.newman
In-Reply-To: <46a038f90706191936m121a94e4x1e59dff4fe217988@mail.gmail.com>
On Wed, 20 Jun 2007, Martin Langhoff wrote:
> A colleage at work is using git to manage code updates to a heavily
> firewalled machine at a client site. In a nutshell, we "push" the
> interesting code to a repo on usb-stick, and we pull from it on the
> client's machine.
>
> We do some coding and testing on that machine, so ocassionally we
> bring some patches back.
>
> Now the working repo on the client machine has started to die with
> "corrupt pack" errors. I am trying to get my hands on the literal
> error messages, and exact software versions installed. Right now all I
> know is that it is SuSE 9 x86, git 1.4.x, cogito .17.x . The error
> appears on git-diff, git-fsck-objects --full
The full exact error message would be highly useful indeed.
> We did bring a copy of the working copy with the "corrupt pack" to our
> office, and here git (v1.5.1 and 1.5.2) thinks it's perfectly well.
>
> So I am a bit puzzled - while we try to get 1.5.x on the client
> machine and see what happens, is there anything that could be causing
> this? Any additional tests that we should run?
Maybe the client machine runs git version < 1.4.2.2, in which case it is
possible that your push created a pack containing delta objects with
offset to base which git versions prior 1.4.2.2 do not understand.
If this is the problem you are facing (the error message should confirm
this) then the easiest solution is to upgrade git on the client.
A quick fix for the client is to set repack.usedeltabaseoffset to
false on the machine where you have git 1.5 installed, then run "git
repack -a -d", and finally copy the pack over to the client repository.
But because you push to a local repository (a mounted USB stick is
considered a local repo) then you don't get to negociate the pack
capabilities of the final destination, and therefore more "bad" delta
objects might sneak in again.
Nicolas
^ permalink raw reply
* Re: Errors install git-1.5.2.2 on 64-bit AIX
From: Dongsheng Song @ 2007-06-20 3:36 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio C Hamano, Git Mailing List
In-Reply-To: <alpine.LFD.0.98.0706192007000.3593@woody.linux-foundation.org>
No "ginstall" or "gnuinstall". I will try compile coreutils-6.9 latter.
2007/6/20, Linus Torvalds <torvalds@linux-foundation.org>:
>
>
> On Wed, 20 Jun 2007, Dongsheng Song wrote:
> >
> > But install failed:
> > [ ... ]
> > Usage: install [-c dira] [-f dirb] [-i] [-m] [-M mode] [-O owner]
> > [-G group] [-S] [-n dirc] [-o] [-s] file [dirx ...]
> > gnumake: *** [install] Error 2
>
> Do you possibly have a "ginstall" somewhere in addition to the GNU make?
>
> If so, just make the "INSTALL" macro in the Makefile point to that instead
> of the (apparently totally broken) regular "install" program on AIX.
>
> Maybe it's called "gnuinstall".
>
> That said, the installation is really just a matter of copying, so you
> *could* just replace the uses of "install" with either "-mkdir" or "cp"
> depending on whether it's used to make sure a directory exists, or to
> actually copy the programs.
>
> Linus
>
^ permalink raw reply
* Re: Debugging strange "corrupt pack" errors on SuSE 9
From: Martin Langhoff @ 2007-06-20 4:17 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Git Mailing List, jonathan.newman
In-Reply-To: <alpine.LFD.0.99.0706192313290.20596@xanadu.home>
On 6/20/07, Nicolas Pitre <nico@cam.org> wrote:
> The full exact error message would be highly useful indeed.
Yes. I haven't seen it first hand either, and the machine is closely
guarded :-/ but I think you are on to something: it's a v1.4.0.
# git --version
> git version 1.4.0
# cat /etc/issue
Welcome to SUSE LINUX Enterprise Server 9 (i586) - Kernel \r (\l).
# uname -a
Linux lhostname 2.6.5-7.244-bigsmp-100hz #3 SMP Sun Aug 13 21:50:35 NZST
2006 i686 i686 i386 GNU/Linux
The errors look like
# git-pull /home/jun/moodle-r2.git
ird-mk2:fromusb
fatal: git-unpack-objects exec failed
fatal: git-unpack-objects died with error code 128
Fetch failure: /home/jun/moodle-r2.git
# cg-update fromusb
Using hard links
Fetching head...
Fetching objects...
Getting pack 445d79a6fe09a3d03489449f63faef0d9e9e2668
which contains 4f4773fb3403f3ec4097ab7c7b1fdec23b9aa924
fatal: corrupted pack file
.git/objects/pack/pack-445d79a6fe09a3d03489449f63faef0d9e9e2668.pack
progress: 2 objects, 0 bytes
cg-fetch: objects fetch failed
> Maybe the client machine runs git version < 1.4.2.2, in which case it is
> possible that your push created a pack containing delta objects with
> offset to base which git versions prior 1.4.2.2 do not understand.
Ouch. We weren't supposed to have non-backwards compatible changes...
> If this is the problem you are facing (the error message should confirm
> this) then the easiest solution is to upgrade git on the client.
Ha ha. Not particularly easy, unfortunately.
> A quick fix for the client is to set repack.usedeltabaseoffset to
> false on the machine where you have git 1.5 installed, then run "git
> repack -a -d", and finally copy the pack over to the client repository.
That'll be a bit easier -- it's a fix we can do on the transfer repo ourselves.
Thanks! I do wonder though -- isn't a backwards-incompatible change
like this worthy of don't we bump core.repositoryformatversion?
> But because you push to a local repository (a mounted USB stick is
> considered a local repo) then you don't get to negociate the pack
> capabilities of the final destination, and therefore more "bad" delta
> objects might sneak in again.
How does that work? So any repo we push _from_ can override (and muck
up) the destination repo, ignoring its config?
That sounds a bit broken - the pack being built for a local
destination should respect the settings of the destination repo.
m
^ permalink raw reply
* Re: Debugging strange "corrupt pack" errors on SuSE 9
From: Martin Langhoff @ 2007-06-20 4:20 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Git Mailing List, jonathan.newman
In-Reply-To: <46a038f90706192117x53420c04o27f05e8fa6c338a5@mail.gmail.com>
On 6/20/07, Martin Langhoff <martin.langhoff@gmail.com> wrote:
> > But because you push to a local repository (a mounted USB stick is
> > considered a local repo) then you don't get to negociate the pack
> > capabilities of the final destination, and therefore more "bad" delta
> > objects might sneak in again.
>
> How does that work? So any repo we push _from_ can override (and muck
> up) the destination repo, ignoring its config?
>
> That sounds a bit broken - the pack being built for a local
> destination should respect the settings of the destination repo.
OTOH, as a workaround, it _should_ work if I force a repack on the
usb-repo after each push, right? It'll wear the USB disk out, waste
human+cpu time and kill some kittens along the way, but it'll do for
the time being.
cheers,
m
^ permalink raw reply
* Re: Debugging strange "corrupt pack" errors on SuSE 9
From: Nicolas Pitre @ 2007-06-20 5:01 UTC (permalink / raw)
To: Martin Langhoff; +Cc: Git Mailing List, jonathan.newman
In-Reply-To: <46a038f90706192117x53420c04o27f05e8fa6c338a5@mail.gmail.com>
On Wed, 20 Jun 2007, Martin Langhoff wrote:
> On 6/20/07, Nicolas Pitre <nico@cam.org> wrote:
> > Maybe the client machine runs git version < 1.4.2.2, in which case it is
> > possible that your push created a pack containing delta objects with
> > offset to base which git versions prior 1.4.2.2 do not understand.
>
> Ouch. We weren't supposed to have non-backwards compatible changes...
Well... to be fair, we should say that your setup is a bit non
conventional.
> > If this is the problem you are facing (the error message should confirm
> > this) then the easiest solution is to upgrade git on the client.
>
> Ha ha. Not particularly easy, unfortunately.
>
> > A quick fix for the client is to set repack.usedeltabaseoffset to
> > false on the machine where you have git 1.5 installed, then run "git
> > repack -a -d", and finally copy the pack over to the client repository.
>
> That'll be a bit easier -- it's a fix we can do on the transfer repo
> ourselves.
>
> Thanks! I do wonder though -- isn't a backwards-incompatible change
> like this worthy of don't we bump core.repositoryformatversion?
The repository hasn't really changed, and even such a version bump
wouldn't help you anyway.
I also note that you're using cogito. And from what I can deduce, it
seems that cogito is simply copying the pack(s) over without further
sanity checks. This only serves to muddy things even further.
If your client machine is so important then you should consider
upgrading to a later git version, and stop using cogito. It will solve
this pack problem, and you'll get added pack sanity checks as a bonus.
Nicolas
^ permalink raw reply
* git-svn strangeness with tags and Squirrelmail repo
From: Martin Langhoff @ 2007-06-20 5:05 UTC (permalink / raw)
To: Git Mailing List
First -- kudos to Eric Wong and company: git-svn can deal with the odd
errors and invalid chunks of XML or UTF-8 that SVN spits at me every
once in a while. And it "just works" in 99% of the situation. Great
stuff.
Now... on to my 1% where it doesn't "just work"... I am trying to get
a working svn to git gateway for Squirrelmail, and getting in trouble
with the tags setting...
git --version
git version 1.5.2.2.238.g7cbf2f2
For starters - it "just works" if I run
git svn init -T trunk -t tags -b branches \
http://squirrelmail.svn.sourceforge.net/svnroot/squirrelmail
git svn fetch
However, every tree (for tags, branches and trunk) is prefixed with
"squirrelmail" and commits to the toplevel "plugins" directory get in
the way. Also, I want the branches and tags to appear in more natural
places, so after init, and before fetch, I change .git/config to say:
[svn-remote "svn"]
url = http://squirrelmail.svn.sourceforge.net/svnroot/squirrelmail
fetch = trunk/squirrelmail:refs/heads/svn/trunk
branches = branches/*/squirrelmail:refs/heads/svn/*
tags = tags/*/squirrelmail:refs/tags/svn/*
and when I do that -- trunk and branches do what I want, but tags
aren't imported anymore. :-/
cheers,
martin
^ permalink raw reply
* Re: Debugging strange "corrupt pack" errors on SuSE 9
From: Martin Langhoff @ 2007-06-20 5:10 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Git Mailing List, jonathan.newman
In-Reply-To: <alpine.LFD.0.99.0706200053520.20596@xanadu.home>
On 6/20/07, Nicolas Pitre <nico@cam.org> wrote:
> Well... to be fair, we should say that your setup is a bit non
> conventional.
You mean this doesn't affect things between 1.4.0 and 1.5.0 on either
end of the git protocol?
> I also note that you're using cogito. And from what I can deduce, it
> seems that cogito is simply copying the pack(s) over without further
> sanity checks. This only serves to muddy things even further.
Yep - we are moving away from cogito; the machine is important, but
it's very awkward to install/upgrade sw on it.
cheers,
m
^ permalink raw reply
* Re: Debugging strange "corrupt pack" errors on SuSE 9
From: Nicolas Pitre @ 2007-06-20 5:12 UTC (permalink / raw)
To: Martin Langhoff; +Cc: Git Mailing List, jonathan.newman
In-Reply-To: <46a038f90706192117x53420c04o27f05e8fa6c338a5@mail.gmail.com>
On Wed, 20 Jun 2007, Martin Langhoff wrote:
> On 6/20/07, Nicolas Pitre <nico@cam.org> wrote:
>
> > But because you push to a local repository (a mounted USB stick is
> > considered a local repo) then you don't get to negociate the pack
> > capabilities of the final destination, and therefore more "bad" delta
> > objects might sneak in again.
>
> How does that work? So any repo we push _from_ can override (and muck
> up) the destination repo, ignoring its config?
>
> That sounds a bit broken - the pack being built for a local
> destination should respect the settings of the destination repo.
But it does! The problem is that _you_ are cheating it by removing the
USB stick and mounting it somewhere else.
Then you're using cogito which bypasses things.
If you use pure git on the client machine, and actually use git-pull or
git-fetch _without_ the -l switch then validation of the updated data
will occur and there would be no way to muck up the destination repo.
Nicolas
^ permalink raw reply
* Re: Debugging strange "corrupt pack" errors on SuSE 9
From: Nicolas Pitre @ 2007-06-20 5:14 UTC (permalink / raw)
To: Martin Langhoff; +Cc: Git Mailing List, jonathan.newman
In-Reply-To: <46a038f90706192120q6c64a854re589f27b3bdbc0d5@mail.gmail.com>
On Wed, 20 Jun 2007, Martin Langhoff wrote:
> OTOH, as a workaround, it _should_ work if I force a repack on the
> usb-repo after each push, right? It'll wear the USB disk out, waste
> human+cpu time and kill some kittens along the way, but it'll do for
> the time being.
Right. And you don't need to use -f with git-repack for it to work.
Nicolas
^ permalink raw reply
* Re: Debugging strange "corrupt pack" errors on SuSE 9
From: Nicolas Pitre @ 2007-06-20 5:17 UTC (permalink / raw)
To: Martin Langhoff; +Cc: Git Mailing List, jonathan.newman
In-Reply-To: <46a038f90706192210t4594cc86he6a2dd6093b9a1f3@mail.gmail.com>
On Wed, 20 Jun 2007, Martin Langhoff wrote:
> On 6/20/07, Nicolas Pitre <nico@cam.org> wrote:
> > Well... to be fair, we should say that your setup is a bit non
> > conventional.
>
> You mean this doesn't affect things between 1.4.0 and 1.5.0 on either
> end of the git protocol?
Exact.
Nicolas
^ permalink raw reply
* Re: Stupid quoting...
From: Junio C Hamano @ 2007-06-20 6:19 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: David Kastrup, git
In-Reply-To: <Pine.LNX.4.64.0706200307070.4059@racer.site>
Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>> I don't see our discourse leading anywhere: the points have been made.
>
> I would really, really, really like to see a solution. Alas, I cannot
> think of one, other than _forcing_ the developers to use ASCII-only
> filenames.
>
> Note that there is no convention yet in Git to state which encoding your
> filenames are supposed to use. And in fact, we already had a fine example
> in git.git why this is particularly difficult. MacOSX is too clever to be
> true, in that it gladly takes filenames in one encoding, but reads those
> filenames out in _another_ encoding. Thus, a "git add <filename>" can well
> end up in git-status saying that a file was deleted, and another file
> (actually the same, but in a different encoding) is untracked.
By the way, the pathname quoting done by "diff" does not even
attempt to tackle that. I already explained why in the thread
so I would not repeat myself.
Having said that, the absolute minimum that needs to be quoted
are double-quote (because it is used by quoting as agreed with
GNU diff/patch maintainer), backslash (used to introduce C-like
quoting), newline and horizontal tab (makes "patch" confused, as
it would make it ambiguous where the pathname ends), so I am not
opposed to a patch that introduces a new mode, probably on by
default _unless_ we are generating --format=email, that does not
quote high byte values. That would solve "My UTF-8 filenames
are unreadable on my terminal" problem.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox