* Re: [PATCH] diff-delta: produce optimal pack data
From: Nicolas Pitre @ 2006-02-24 20:19 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Carl Baldwin, Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.64.0602241152290.22647@g5.osdl.org>
On Fri, 24 Feb 2006, Linus Torvalds wrote:
> The other thing to look at is "max_size": right now it initializes that to
> "size / 2 - 20", which just says that we don't ever want a delta that is
> larger than about half the result (plus the 20 byte overhead for pointing
> to the thing we delta against). Again, if you feel that normal compression
> compresses better than half, you could try changing that to
>
> ..
> max_size = size / 4 - 20;
> ..
Like I mentioned, max_size should also be caped with the deflated
undeltified object
size. This value is easy to get since plain objects are already
deflated.
> NOTE! Every SINGLE one of those heuristics are just totally made up by
> yours truly, and have no testing behind them. They're more of the type
> "that sounds about right" than "this is how it must be". As mentioned,
> Nico has already been playing with the heuristics - but he wanted better
> packs, not better CPU usage, so he went the other way from what you would
> want to try..
Actually it's a good balance I'm after.
Using 30% more CPU for 10% smaller packs is OK I'd say.
Using 100 times the CPU for 50% saving on only one particular delta is
not acceptable.
And using more than one hour for 200MB of data with the current window
default is not acceptable either.
Nicolas
^ permalink raw reply
* Re: [PATCH] diff-delta: produce optimal pack data
From: Linus Torvalds @ 2006-02-24 20:02 UTC (permalink / raw)
To: Carl Baldwin; +Cc: Nicolas Pitre, Junio C Hamano, git
In-Reply-To: <20060224192354.GC387@hpsvcnb.fc.hp.com>
On Fri, 24 Feb 2006, Carl Baldwin wrote:
>
> I meant that there is no benefit in disk space usage. Packing may
> actually increase my disk space usage in this case. Refer to what I
> said about experimentally running gzip and xdelta on the files
> independantly of git.
Yes. The deltas tend to compress a lot less well than "normal" files.
> I see what you're saying about this data reuse helping to speed up
> subsequent cloning operations. However, if packing takes this long and
> doesn't give me any disk space savings then I don't want to pay the very
> heavy price of packing these files even the first time nor do I want to
> pay the price incrementally.
I would look at tuning the heuristics in "try_delta()" (pack-objects.c) a
bit. That's the place that decides whether to even bother trying to make a
delta, and how big a delta is acceptable. For example, looking at them, I
already see one bug:
..
sizediff = oldsize > size ? oldsize - size : size - oldsize;
if (sizediff > size / 8)
return -1;
..
we really should compare sizediff to the _smaller_ of the two sizes, and
skip the delta if the difference in sizes is bound to be bigger than that.
However, the "size / 8" thing isn't a very strict limit anyway, so this
probably doesn't matter (and I think Nico already removed it as part of
his patches: the heuristic can make us avoid some deltas that would be
ok).
The other thing to look at is "max_size": right now it initializes that to
"size / 2 - 20", which just says that we don't ever want a delta that is
larger than about half the result (plus the 20 byte overhead for pointing
to the thing we delta against). Again, if you feel that normal compression
compresses better than half, you could try changing that to
..
max_size = size / 4 - 20;
..
or something like that instead (but then you need to check that it's still
positive - otherwise the comparisons with unsigned later on are screwed
up. Right now that value is guaranteed to be positive if only because we
already checked
..
if (size < 50)
return -1;
..
earlier).
NOTE! Every SINGLE one of those heuristics are just totally made up by
yours truly, and have no testing behind them. They're more of the type
"that sounds about right" than "this is how it must be". As mentioned,
Nico has already been playing with the heuristics - but he wanted better
packs, not better CPU usage, so he went the other way from what you would
want to try..
Linus
^ permalink raw reply
* git-mailinfo doesn't get installed any more
From: Tony Luck @ 2006-02-24 20:06 UTC (permalink / raw)
To: Git Mailing List
Periodically after I upgrade git I do an "ls -lrt /usr/local/bin" to
find stray old binaries that aren't part of git anymore. I was
a bit surprised to see that git-mailinfo had a mod-time a bit
older than the rest of git ... and looking at the Makefile it looks
like it got dropped in some rearrangement.
Two things:
1) Can someone put it back please, git-applymbox is very unhappy
without it.
2) What's the cute 1-line git way to see when this was broken. I'm
guessing that it involves using a --pickaxe.
Thanks
-Tony
^ permalink raw reply
* Re: [PATCH] diff-delta: produce optimal pack data
From: Nicolas Pitre @ 2006-02-24 20:02 UTC (permalink / raw)
To: Carl Baldwin; +Cc: Junio C Hamano, git
In-Reply-To: <20060224192354.GC387@hpsvcnb.fc.hp.com>
On Fri, 24 Feb 2006, Carl Baldwin wrote:
> I see what you're saying about this data reuse helping to speed up
> subsequent cloning operations. However, if packing takes this long and
> doesn't give me any disk space savings then I don't want to pay the very
> heavy price of packing these files even the first time nor do I want to
> pay the price incrementally.
Of course. There is admitedly a problem here. I'm just abusing a bit
of your time to properly identify its parameters.
> The most I would tolerate for the first pack is a few seconds. The most
> I would tolerate for any incremental pack is about 1 second.
Well that is probably a bit tight. Ideally it should be linear with the
size of the data set to process. If you have 10 files 10MB each it
should take about the same time to pack than 10000 files of 10KB each.
Of course incrementally packing one additional 10MB file might take more
than a second although it is only one file.
> BTW, git repack has been going for 30 minutes and has packed 4/36
> objects. :-)
Pathetic.
> I think the right answer would be for git to avoid trying to pack files
> like this. Junio mentioned something like this in his message.
Yes. First we could add an additional parameter to the repacking
strategy which is the undeltified but deflated size of an object. That
would prevent any deltas to become bigger than the simply deflated
version.
Remains the delta performance issue. I think I know what the problem
is. I'm not sure I know what the best solution would be though. The
patological data set is easy to identify quickly and one strategy might
simply to bail out early when it happens and therefore not attempt any
delta.
However, if you could let me play with two samples of your big file I'd
be grateful. If so I'd like to make git work well with your data set
too which is not that uncommon after all.
Nicolas
^ permalink raw reply
* Re: git-annotate efficiency
From: Ryan Anderson @ 2006-02-24 19:42 UTC (permalink / raw)
To: Morten Welinder; +Cc: GIT Mailing List
In-Reply-To: <118833cc0602241000p4e4c8017u3e3afe76fbbd75a4@mail.gmail.com>
On Fri, Feb 24, 2006 at 01:00:24PM -0500, Morten Welinder wrote:
> It looks like handle_rev is seeing the same revisions over and over again.
> I don't know why that would be, but the following patch just skips dups.
> I have no idea if it is right, though.
Merges.
a--b--c--d--f
\-g--h--/
It would do f,d,c,b,a + f,h,g,b,a
So yes, this fix is correct, and Junio, I'll be doing some changes this
weekend and send it along with a few other things.
(On a medium-sized test tree at work with 3500 commits in the tree, 37
on the main Makefile (6 merges), this cuts the annotate time from 10s to a little
over 2, so it's, umm, very worthwhile.)
>
> Morten
>
>
> diff --git a/git-annotate.perl b/git-annotate.perl
> index 3800c46..a5e2d86 100755
> --- a/git-annotate.perl
> +++ b/git-annotate.perl
> @@ -117,7 +117,10 @@ sub init_claim {
>
> sub handle_rev {
> my $i = 0;
> + my %seen = ();
> while (my $rev = shift @revqueue) {
> + next if $seen{$rev};
> + $seen{$rev} = 1;
>
> my %revinfo = git_commit_info($rev);
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
Ryan Anderson
sometimes Pug Majere
^ permalink raw reply
* Re: Removal of "--merge-order"?
From: Junio C Hamano @ 2006-02-24 19:25 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <Pine.LNX.4.64.0602240824110.3771@g5.osdl.org>
Linus Torvalds <torvalds@osdl.org> writes:
> In other words, I'd really prefer if it was gone. Some of the things I
> might do to git-rev-list would be much simpler if I didn't have to worry
> about merge-order, and the way it interfaces with the rest of
> git-rev-list.
>
> Comments?
>
> Linus
I am really glad you brought it up. I would not miss it at all.
^ permalink raw reply
* Re: [PATCH] diff-delta: produce optimal pack data
From: Carl Baldwin @ 2006-02-24 19:23 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.64.0602241350190.31162@localhost.localdomain>
On Fri, Feb 24, 2006 at 01:57:20PM -0500, Nicolas Pitre wrote:
> On Fri, 24 Feb 2006, Carl Baldwin wrote:
>
> > My version is 1.2.1. I have not been following your work. When was
> > pack data reuse introduced?
>
> Try out version 1.2.3.
I'm on it now.
> > From where can I obtain your delta patches?
>
> Forget them for now -- they won't help you.
Ah, I have been looking at your patches and clearly they will not help.
> > There is really no opportunity for pack-data reuse in this case. The
> > repository had never been packed or cloned in the first place. As I
> > said, I do not intend to pack these binary files at all since there is
> > no benefit in this case.
>
> Yes there is, as long as you have version 1.2.3. The clone logic will
> simply reuse already packed data without attempting to recompute it.
I meant that there is no benefit in disk space usage. Packing may
actually increase my disk space usage in this case. Refer to what I
said about experimentally running gzip and xdelta on the files
independantly of git.
I see what you're saying about this data reuse helping to speed up
subsequent cloning operations. However, if packing takes this long and
doesn't give me any disk space savings then I don't want to pay the very
heavy price of packing these files even the first time nor do I want to
pay the price incrementally.
The most I would tolerate for the first pack is a few seconds. The most
I would tolerate for any incremental pack is about 1 second.
BTW, git repack has been going for 30 minutes and has packed 4/36
objects. :-)
> I think you really should try git version 1.2.3 with a packed
> repository. It might handle your special case just fine.
No, not when I'm not willing to pay the price to pack even once. This
isn't a case where I have one such repository and 'once its been packed
then its packed'. This is only one example of such a repository. I am
looking for a process for revisioning this type of data that will be
used over and over. Git may not be the answer here but it sure is
looking good in many other ways.
I think the right answer would be for git to avoid trying to pack files
like this. Junio mentioned something like this in his message.
Thanks for your input.
Cheers,
Carl
--
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Carl Baldwin RADCAD (R&D CAD)
Hewlett Packard Company
MS 88 work: 970 898-1523
3404 E. Harmony Rd. work: Carl.N.Baldwin@hp.com
Fort Collins, CO 80525 home: Carl@ecBaldwin.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
^ permalink raw reply
* Re: git-annotate efficiency
From: Randal L. Schwartz @ 2006-02-24 19:06 UTC (permalink / raw)
To: Morten Welinder; +Cc: GIT Mailing List
In-Reply-To: <118833cc0602241000p4e4c8017u3e3afe76fbbd75a4@mail.gmail.com>
>>>>> "Morten" == Morten Welinder <mwelinder@gmail.com> writes:
Morten> It looks like handle_rev is seeing the same revisions over and over again.
Morten> I don't know why that would be, but the following patch just skips dups.
Morten> I have no idea if it is right, though.
Morten> Morten
Morten> diff --git a/git-annotate.perl b/git-annotate.perl
Morten> index 3800c46..a5e2d86 100755
Morten> --- a/git-annotate.perl
Morten> +++ b/git-annotate.perl
Morten> @@ -117,7 +117,10 @@ sub init_claim {
Morten> sub handle_rev {
Morten> my $i = 0;
Morten> + my %seen = ();
Morten> while (my $rev = shift @revqueue) {
Morten> + next if $seen{$rev};
Morten> + $seen{$rev} = 1;
Morten> my %revinfo = git_commit_info($rev);
The traditional idiom for that is
next if $seen{$rev}++;
--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
^ permalink raw reply
* Re: [PATCH] diff-delta: produce optimal pack data
From: Nicolas Pitre @ 2006-02-24 19:03 UTC (permalink / raw)
To: Carl Baldwin; +Cc: Junio C Hamano, git
In-Reply-To: <20060224184934.GA387@hpsvcnb.fc.hp.com>
On Fri, 24 Feb 2006, Carl Baldwin wrote:
> I've updated to a very current master branch. This seems to include the
> pack data reuse stuff. I've not made an attempt yet to apply your delta
> patches.
>
> git-repack quickly gets up to 5% (2/36) and hangs there. I'll let it
> run for a while just to see how far it claims to get. I'm not hopeful.
It should complete sometimes, probably after the same amount of time
needed by your previous clone attempt. But after that any clone
operation should be quick. This is clearly unacceptable but at least
with the pack data reuse you should suffer only once for the initial
repack.
> Maybe your patches can help?
No. They actually make things worse performance wise, much worse in
some special cases.
Is it possible for me to have access to 2 consecutive versions of your
big binary file?
Nicolas
^ permalink raw reply
* Re: [PATCH] diff-delta: produce optimal pack data
From: Nicolas Pitre @ 2006-02-24 18:57 UTC (permalink / raw)
To: Carl Baldwin; +Cc: Junio C Hamano, git
In-Reply-To: <20060224183554.GA31247@hpsvcnb.fc.hp.com>
On Fri, 24 Feb 2006, Carl Baldwin wrote:
> My version is 1.2.1. I have not been following your work. When was
> pack data reuse introduced?
Try out version 1.2.3.
> From where can I obtain your delta patches?
Forget them for now -- they won't help you.
> There is really no opportunity for pack-data reuse in this case. The
> repository had never been packed or cloned in the first place. As I
> said, I do not intend to pack these binary files at all since there is
> no benefit in this case.
Yes there is, as long as you have version 1.2.3. The clone logic will
simply reuse already packed data without attempting to recompute it.
> The delta patches may help but I can't say for sure since I don't know
> anything about them.
They (actually the last one) might help reduce the size of resulting
packs but it currently has performance problems with some patological
data sets.
I think you really should try git version 1.2.3 with a packed
repository. It might handle your special case just fine.
Nicolas
^ permalink raw reply
* Re: [PATCH] diff-delta: produce optimal pack data
From: Carl Baldwin @ 2006-02-24 18:49 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.64.0602241252300.31162@localhost.localdomain>
I've updated to a very current master branch. This seems to include the
pack data reuse stuff. I've not made an attempt yet to apply your delta
patches.
git-repack quickly gets up to 5% (2/36) and hangs there. I'll let it
run for a while just to see how far it claims to get. I'm not hopeful.
Maybe your patches can help?
Carl
On Fri, Feb 24, 2006 at 12:56:04PM -0500, Nicolas Pitre wrote:
> On Fri, 24 Feb 2006, Carl Baldwin wrote:
>
> > Junio,
> >
> > This message came to me at exactly the right time. Yesterday I was
> > exploring using git as the content storage back-end for some binary
> > files. Up until now I've only used it for software projects.
> >
> > I found the largest RCS file that we had in our current back-end. It
> > contained twelve versions of a binary file. Each version averaged about
> > 20 MB. The ,v file from RCS was about 250MB. I did some experiments on
> > these binary files.
> >
> > First, gzip consistantly is able to compress these files to about 10%
> > their original size. So, they are quite inflated. Second, xdelta would
> > produce a delta between two neighboring revisions of about 2.5MB in size
> > that would compress down to about 2MB. (about the same size as the next
> > revision compressed without deltification so packing is ineffective
> > here).
> >
> > I added these 12 revisions to several version control back-ends
> > including subversion and git. Git produced a much smaller repository
> > size than the others simply due to the compression that it applies to
> > objects. It also was at least as fast as the others.
> >
> > The problem came when I tried to clone this repository.
> > git-pack-objects chewed on these 12 revisions for over an hour before I
> > finally interrupted it. As far as I could tell, it hadn't made much
> > progress.
>
> I must ask if you had applied my latest delta patches?
>
> Also did you use a recent version of git that implements pack data
> reuse?
>
>
> Nicolas
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Carl Baldwin RADCAD (R&D CAD)
Hewlett Packard Company
MS 88 work: 970 898-1523
3404 E. Harmony Rd. work: Carl.N.Baldwin@hp.com
Fort Collins, CO 80525 home: Carl@ecBaldwin.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
^ permalink raw reply
* Re: [PATCH] diff-delta: produce optimal pack data
From: Carl Baldwin @ 2006-02-24 18:35 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.64.0602241252300.31162@localhost.localdomain>
On Fri, Feb 24, 2006 at 12:56:04PM -0500, Nicolas Pitre wrote:
My version is 1.2.1. I have not been following your work. When was
pack data reuse introduced? From where can I obtain your delta patches?
There is really no opportunity for pack-data reuse in this case. The
repository had never been packed or cloned in the first place. As I
said, I do not intend to pack these binary files at all since there is
no benefit in this case.
The delta patches may help but I can't say for sure since I don't know
anything about them. Let me know where I can get them.
Carl
>
> I must ask if you had applied my latest delta patches?
>
> Also did you use a recent version of git that implements pack data
> reuse?
>
>
> Nicolas
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Carl Baldwin RADCAD (R&D CAD)
Hewlett Packard Company
MS 88 work: 970 898-1523
3404 E. Harmony Rd. work: Carl.N.Baldwin@hp.com
Fort Collins, CO 80525 home: Carl@ecBaldwin.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
^ permalink raw reply
* Re: Removal of "--merge-order"?
From: Randy.Dunlap @ 2006-02-24 18:10 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Randy.Dunlap, Junio C Hamano, Git Mailing List
In-Reply-To: <Pine.LNX.4.64.0602240957430.22647@g5.osdl.org>
On Fri, 24 Feb 2006, Linus Torvalds wrote:
>
>
> On Fri, 24 Feb 2006, Randy.Dunlap wrote:
> >
> > Other than Ryan's reply, I found 2 users in a quick search,
> > but they have already stated that they are willing to change, so I
> > don't see objections unless someone else comes forward.
>
> One thing we could do - and might be simpler - is to make the merge-order
> thing be a post-processing phase of git-rev-list.
>
> IOW, instead of
>
> git-rev-list --merge-order
>
> we could perhaps do
>
> git-rev-list --parents [--topo-order?] | git-merge-order
>
> so that the merge-order code wouldn't impact git-rev-list itself.
Makes sense to me... thanks.
But even that may not be needed if noone else really needs it.
> As it is, the merge-order code ends up hooking into the "process_commit"
> thing (and thus to "filter_commit" which does the parent rewriting, and
> then show_commit), which makes it harder to work with.
>
> Now, rev-list.c is not the biggest file (apply.c is about twice the size),
> but in many ways it's the most complex one by far. It's also the most
> performance-critical one, and the one that it would be really nice if we
> were to be able to libify it.
>
> For example, instead of the horrid scriping language, I _think_ I could
> almost libify it by just hooking into "show_commit", and using a callback
> function for that (and then the stand-alone program would just make the
> callback function be one that prints out the commit).
>
> With some care, we might be able to make things like "git diff" be small C
> programs (or, more likely, to save space and not replicate the binaries
> many times - make the "git" binary able to do all the simple things on its
> own: "git-diff" would be just a link to "git").
>
> That would possibly be a simpler way to get away from using nonportable
> scripts. Plain C really does remain one of the most portable things out
> there.
--
~Randy
^ permalink raw reply
* Re: Removal of "--merge-order"?
From: Linus Torvalds @ 2006-02-24 18:07 UTC (permalink / raw)
To: Randy.Dunlap; +Cc: Junio C Hamano, Git Mailing List
In-Reply-To: <Pine.LNX.4.58.0602240942520.7894@shark.he.net>
On Fri, 24 Feb 2006, Randy.Dunlap wrote:
>
> Other than Ryan's reply, I found 2 users in a quick search,
> but they have already stated that they are willing to change, so I
> don't see objections unless someone else comes forward.
One thing we could do - and might be simpler - is to make the merge-order
thing be a post-processing phase of git-rev-list.
IOW, instead of
git-rev-list --merge-order
we could perhaps do
git-rev-list --parents [--topo-order?] | git-merge-order
so that the merge-order code wouldn't impact git-rev-list itself.
As it is, the merge-order code ends up hooking into the "process_commit"
thing (and thus to "filter_commit" which does the parent rewriting, and
then show_commit), which makes it harder to work with.
Now, rev-list.c is not the biggest file (apply.c is about twice the size),
but in many ways it's the most complex one by far. It's also the most
performance-critical one, and the one that it would be really nice if we
were to be able to libify it.
For example, instead of the horrid scriping language, I _think_ I could
almost libify it by just hooking into "show_commit", and using a callback
function for that (and then the stand-alone program would just make the
callback function be one that prints out the commit).
With some care, we might be able to make things like "git diff" be small C
programs (or, more likely, to save space and not replicate the binaries
many times - make the "git" binary able to do all the simple things on its
own: "git-diff" would be just a link to "git").
That would possibly be a simpler way to get away from using nonportable
scripts. Plain C really does remain one of the most portable things out
there.
Linus
^ permalink raw reply
* Re: git-annotate efficiency
From: Morten Welinder @ 2006-02-24 18:00 UTC (permalink / raw)
To: GIT Mailing List
In-Reply-To: <118833cc0602240737i42acdc90sb8f93dde1a1bc035@mail.gmail.com>
It looks like handle_rev is seeing the same revisions over and over again.
I don't know why that would be, but the following patch just skips dups.
I have no idea if it is right, though.
Morten
diff --git a/git-annotate.perl b/git-annotate.perl
index 3800c46..a5e2d86 100755
--- a/git-annotate.perl
+++ b/git-annotate.perl
@@ -117,7 +117,10 @@ sub init_claim {
sub handle_rev {
my $i = 0;
+ my %seen = ();
while (my $rev = shift @revqueue) {
+ next if $seen{$rev};
+ $seen{$rev} = 1;
my %revinfo = git_commit_info($rev);
^ permalink raw reply related
* Re: [PATCH] diff-delta: produce optimal pack data
From: Nicolas Pitre @ 2006-02-24 17:56 UTC (permalink / raw)
To: Carl Baldwin; +Cc: Junio C Hamano, git
In-Reply-To: <20060224174422.GA13367@hpsvcnb.fc.hp.com>
On Fri, 24 Feb 2006, Carl Baldwin wrote:
> Junio,
>
> This message came to me at exactly the right time. Yesterday I was
> exploring using git as the content storage back-end for some binary
> files. Up until now I've only used it for software projects.
>
> I found the largest RCS file that we had in our current back-end. It
> contained twelve versions of a binary file. Each version averaged about
> 20 MB. The ,v file from RCS was about 250MB. I did some experiments on
> these binary files.
>
> First, gzip consistantly is able to compress these files to about 10%
> their original size. So, they are quite inflated. Second, xdelta would
> produce a delta between two neighboring revisions of about 2.5MB in size
> that would compress down to about 2MB. (about the same size as the next
> revision compressed without deltification so packing is ineffective
> here).
>
> I added these 12 revisions to several version control back-ends
> including subversion and git. Git produced a much smaller repository
> size than the others simply due to the compression that it applies to
> objects. It also was at least as fast as the others.
>
> The problem came when I tried to clone this repository.
> git-pack-objects chewed on these 12 revisions for over an hour before I
> finally interrupted it. As far as I could tell, it hadn't made much
> progress.
I must ask if you had applied my latest delta patches?
Also did you use a recent version of git that implements pack data
reuse?
Nicolas
^ permalink raw reply
* Re: Removal of "--merge-order"?
From: Randy.Dunlap @ 2006-02-24 17:47 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Randy.Dunlap, Junio C Hamano, Git Mailing List
In-Reply-To: <Pine.LNX.4.64.0602240918030.3771@g5.osdl.org>
On Fri, 24 Feb 2006, Linus Torvalds wrote:
>
>
> On Fri, 24 Feb 2006, Randy.Dunlap wrote:
> >
> > I'm just a lowly user, but I see people trying to export git
> > trees to other SCMs, and they seem to prefer merge-order.
> > This is your chance to correct me about:
> > (a) how I am wrong; (b) how they are wrong. 8;)
>
> Well, I didn't even realize anybody at all was using it. I've never seen
> any mention of it, and considering how ungodly slow it is, I would have
> expected somebody to pipe up about it..
>
> I did a google search for "git" and "merge-order", and the only actual use
> (as opposed to mention in a man-page) I found in the 20 hits google showed
> was an old version of gitk.
>
> > I've heard/seen you say that merge-order is not interesting,
> > but I still believe that *your* merge order of the Linux kernel
> > tree is almost all that people really care about.
>
> Could you actually point to somebody using it? They're hiding it well.
Other than Ryan's reply, I found 2 users in a quick search,
but they have already stated that they are willing to change, so I
don't see objections unless someone else comes forward.
(Martin Langhoff, archimport)
http://marc.theaimsgroup.com/?l=git&m=112682069025547&w=2
Jon Seymour:
http://marc.theaimsgroup.com/?l=git&m=112998877717814&w=2
> > Apparently I needed to go to LCA to hear you discuss git.
>
> I certainly never delved into any of that..
Darn.
--
~Randy
^ permalink raw reply
* Re: [PATCH] diff-delta: produce optimal pack data
From: Carl Baldwin @ 2006-02-24 17:44 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Nicolas Pitre, git
In-Reply-To: <7v4q2pf8fq.fsf@assigned-by-dhcp.cox.net>
Junio,
This message came to me at exactly the right time. Yesterday I was
exploring using git as the content storage back-end for some binary
files. Up until now I've only used it for software projects.
I found the largest RCS file that we had in our current back-end. It
contained twelve versions of a binary file. Each version averaged about
20 MB. The ,v file from RCS was about 250MB. I did some experiments on
these binary files.
First, gzip consistantly is able to compress these files to about 10%
their original size. So, they are quite inflated. Second, xdelta would
produce a delta between two neighboring revisions of about 2.5MB in size
that would compress down to about 2MB. (about the same size as the next
revision compressed without deltification so packing is ineffective
here).
I added these 12 revisions to several version control back-ends
including subversion and git. Git produced a much smaller repository
size than the others simply due to the compression that it applies to
objects. It also was at least as fast as the others.
The problem came when I tried to clone this repository.
git-pack-objects chewed on these 12 revisions for over an hour before I
finally interrupted it. As far as I could tell, it hadn't made much
progress.
My other complaint was that git prune ran slow (~8 seconds on my very
fast machine with fast disk access) on a repository with only these
twelve revisions in it (37 total objects in the object store). This is
because 'git prune' actually ends up running fsck on all of the objects
which verifies the sha1 of each object. This seems like a lot of work
just to prune unwanted objects. What would you say to a --fast option
to git-prune that would avoid most of what fsck does including verifying
sha1 for each object?
Anyway, that was a tangent. I looked into to overriding the --depth
option to git-pack-objects and set it to 0. However, this isn't
trivial. git-pack-objects is never called directly by the user. It is
only called through things like 'git clone', 'git push' and 'git
repack'. What do you think about this? Could we add a configuration
option that could be set for the repository? Something smarter like
what you suggest where git would pack small text files but give up on
large binaries would be optimal.
I've already determined that packing a repository with this type of
largish binary file doesn't do any good but there doesn't seem to be a
way to avoid packing when doing network operations.
Thoughts?
Carl
On Fri, Feb 24, 2006 at 12:49:13AM -0800, Junio C Hamano wrote:
> Nicolas Pitre <nico@cam.org> writes:
>
> > Indexing based on adler32 has a match precision based on the block size
> > (currently 16). Lowering the block size would produce smaller deltas
> > but the indexing memory and computing cost increases significantly.
>
> Indeed.
>
> I had this patch in my personal tree for a while. I was
> wondring why sometimes progress indication during "Deltifying"
> stage stops for literally several seconds, or more.
>
> In Linux 2.6 repository, these object pairs take forever to
> delta.
>
> blob 9af06ba723df75fed49f7ccae5b6c9c34bc5115f ->
> blob dfc9cd58dc065d17030d875d3fea6e7862ede143
> size (491102 -> 496045)
> 58 seconds
>
> blob 4917ec509720a42846d513addc11cbd25e0e3c4f ->
> blob dfc9cd58dc065d17030d875d3fea6e7862ede143
> size (495831 -> 496045)
> 64 seconds
>
> Admittedly, these are *BAD* input samples (a binary firmware
> blob with many similar looking ", 0x" sequences). I can see
> that trying to reuse source materials really hard would take
> significant computation.
>
> However, this is simply unacceptable.
>
> The new algoritm takes 58 seconds to produce 136000 bytes of
> delta, while the old takes 0.25 seconds to produce 248899 (I am
> using the test-delta program in git.git distribution). The
> compression ratio is significantly better, but this is unusable
> even for offline archival use (remember, pack delta selection
> needs to do window=10 such deltification trials to come up with
> the best delta, so you are spending 10 minutes to save 100k from
> one oddball blob), let alone on-the-fly pack generation for
> network transfer.
>
> Maybe we would want two implementation next to each other, and
> internally see if it is taking too much cycles compared to the
> input size then switch to cheaper version?
>
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Carl Baldwin RADCAD (R&D CAD)
Hewlett Packard Company
MS 88 work: 970 898-1523
3404 E. Harmony Rd. work: Carl.N.Baldwin@hp.com
Fort Collins, CO 80525 home: Carl@ecBaldwin.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
^ permalink raw reply
* Re: Removal of "--merge-order"?
From: Ryan Anderson @ 2006-02-24 17:32 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Randy.Dunlap, Junio C Hamano, Git Mailing List
In-Reply-To: <Pine.LNX.4.64.0602240918030.3771@g5.osdl.org>
On Fri, Feb 24, 2006 at 09:23:24AM -0800, Linus Torvalds wrote:
> On Fri, 24 Feb 2006, Randy.Dunlap wrote:
> >
> > I'm just a lowly user, but I see people trying to export git
> > trees to other SCMs, and they seem to prefer merge-order.
> > This is your chance to correct me about:
> > (a) how I am wrong; (b) how they are wrong. 8;)
>
> Well, I didn't even realize anybody at all was using it. I've never seen
> any mention of it, and considering how ungodly slow it is, I would have
> expected somebody to pipe up about it..
>
> I did a google search for "git" and "merge-order", and the only actual use
> (as opposed to mention in a man-page) I found in the 20 hits google showed
> was an old version of gitk.
http://www.gelato.unsw.edu.au/archives/git/0511/12965.html
But topo-order would probably work as well, the default ordering just
didn't work correctly in my tests.
Certainly not a case that votes *against* removal, just noting an actual
user at one point.
--
Ryan Anderson
sometimes Pug Majere
^ permalink raw reply
* Re: Removal of "--merge-order"?
From: Linus Torvalds @ 2006-02-24 17:23 UTC (permalink / raw)
To: Randy.Dunlap; +Cc: Junio C Hamano, Git Mailing List
In-Reply-To: <Pine.LNX.4.58.0602240840520.7894@shark.he.net>
On Fri, 24 Feb 2006, Randy.Dunlap wrote:
>
> I'm just a lowly user, but I see people trying to export git
> trees to other SCMs, and they seem to prefer merge-order.
> This is your chance to correct me about:
> (a) how I am wrong; (b) how they are wrong. 8;)
Well, I didn't even realize anybody at all was using it. I've never seen
any mention of it, and considering how ungodly slow it is, I would have
expected somebody to pipe up about it..
I did a google search for "git" and "merge-order", and the only actual use
(as opposed to mention in a man-page) I found in the 20 hits google showed
was an old version of gitk.
> I've heard/seen you say that merge-order is not interesting,
> but I still believe that *your* merge order of the Linux kernel
> tree is almost all that people really care about.
Could you actually point to somebody using it? They're hiding it well.
> Apparently I needed to go to LCA to hear you discuss git.
I certainly never delved into any of that..
Linus
^ permalink raw reply
* Re: Removal of "--merge-order"?
From: Randy.Dunlap @ 2006-02-24 16:53 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio C Hamano, Git Mailing List
In-Reply-To: <Pine.LNX.4.64.0602240824110.3771@g5.osdl.org>
On Fri, 24 Feb 2006, Linus Torvalds wrote:
>
> I just tested it again, and
>
> git-rev-list --merge-order HEAD
>
> takes an inordinate amount of time:
>
> real 5m1.139s
> user 4m59.504s
> sys 0m1.220s
That's too bad.
> and that's on a reasonably fast machine (not my fastest, but no slouch by
> any measure - my fastest machine I'm not allowed to really benchmark
> publicly ;)
>
> It may be a cool algorithm, but it's essentially useless on any bigger
> tree. And nobody uses it, since "--topo-order" gives the guarantees that
> people really care about, and finishes in 0.537 seconds on the same
> machine with the same tree.
>
> It also depends on the openssh "bignum" stuff, which means that any
> machine where we just rely on our own SHA1 implementation and don't use
> openssh doesn't have the flag anyway.
>
> In other words, I'd really prefer if it was gone. Some of the things I
> might do to git-rev-list would be much simpler if I didn't have to worry
> about merge-order, and the way it interfaces with the rest of
> git-rev-list.
>
> Comments?
I'm just a lowly user, but I see people trying to export git
trees to other SCMs, and they seem to prefer merge-order.
This is your chance to correct me about:
(a) how I am wrong; (b) how they are wrong. 8;)
I've heard/seen you say that merge-order is not interesting,
but I still believe that *your* merge order of the Linux kernel
tree is almost all that people really care about.
Apparently I needed to go to LCA to hear you discuss git.
--
~Randy
^ permalink raw reply
* Removal of "--merge-order"?
From: Linus Torvalds @ 2006-02-24 16:32 UTC (permalink / raw)
To: Junio C Hamano, Git Mailing List
I just tested it again, and
git-rev-list --merge-order HEAD
takes an inordinate amount of time:
real 5m1.139s
user 4m59.504s
sys 0m1.220s
and that's on a reasonably fast machine (not my fastest, but no slouch by
any measure - my fastest machine I'm not allowed to really benchmark
publicly ;)
It may be a cool algorithm, but it's essentially useless on any bigger
tree. And nobody uses it, since "--topo-order" gives the guarantees that
people really care about, and finishes in 0.537 seconds on the same
machine with the same tree.
It also depends on the openssh "bignum" stuff, which means that any
machine where we just rely on our own SHA1 implementation and don't use
openssh doesn't have the flag anyway.
In other words, I'd really prefer if it was gone. Some of the things I
might do to git-rev-list would be much simpler if I didn't have to worry
about merge-order, and the way it interfaces with the rest of
git-rev-list.
Comments?
Linus
^ permalink raw reply
* Re: gitview: Use monospace font to draw the branch and tag name
From: Aneesh Kumar @ 2006-02-24 16:29 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vr75tc8gj.fsf@assigned-by-dhcp.cox.net>
On 2/24/06, Junio C Hamano <junkio@cox.net> wrote:
> "Aneesh Kumar K.V" <aneesh.kumar@gmail.com> writes:
>
> > This patch address the below:
> > Use monospace font to draw branch and tag name
> > set the font size to 13.
>
> I have an impression that hardcoding UI policy like this is
> generally frowned upon.
But with that changes branch and the tag name looks neat.
May be down the line we can add a prefernce tag that will allow the
user to change all these hardcoded values.
-aneesh
^ permalink raw reply
* gitview: Fix the graph display
From: Aneesh Kumar K.V @ 2006-02-24 16:27 UTC (permalink / raw)
To: git, Junio C Hamano
[-- Attachment #1: Type: text/plain, Size: 0 bytes --]
[-- Attachment #2: 0005-gitview-Fix-the-graph-display.txt --]
[-- Type: text/plain, Size: 1574 bytes --]
Subject: gitview: Fix the graph display .
This fix all the known issue with the graph display
The bug need to be explained graphically
|
a
This line need not be there ---->| \
b |
| /
c
c is parent of a and all a,b and c are placed on the same line and b is child of c
With my last checkin I added a seperate line to indicate that a is
connected to c. But then we had the line connecting a and b which should
not be ther. This changes fixes the same bug
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@gmail.com>
---
contrib/gitview/gitview | 6 ++++--
1 files changed, 4 insertions(+), 2 deletions(-)
d4da5f1243c47322ede9dae2a65098cbc7e9ecb5
diff --git a/contrib/gitview/gitview b/contrib/gitview/gitview
index 2cde71e..4e3847d 100755
--- a/contrib/gitview/gitview
+++ b/contrib/gitview/gitview
@@ -938,8 +938,10 @@ class GitView:
def draw_incomplete_line(self, sha1, node_pos, out_line, in_line, index):
for idx, pos in enumerate(self.incomplete_line[sha1]):
if(pos == node_pos):
- out_line.append((pos,
- pos+0.5, self.colours[sha1]))
+ #remove the straight line and add a slash
+ if ((pos, pos, self.colours[sha1]) in out_line):
+ out_line.remove((pos, pos, self.colours[sha1]))
+ out_line.append((pos, pos+0.5, self.colours[sha1]))
self.incomplete_line[sha1][idx] = pos = pos+0.5
try:
next_commit = self.commits[index+1]
--
1.2.3.g2cf3-dirty
^ permalink raw reply related
* gitview: Code cleanup
From: Aneesh Kumar K.V @ 2006-02-24 16:19 UTC (permalink / raw)
To: git, Junio C Hamano
[-- Attachment #1: Type: text/plain, Size: 0 bytes --]
[-- Attachment #2: 0004-gitview-Code-cleanup.txt --]
[-- Type: text/plain, Size: 3109 bytes --]
Subject: gitview: Code cleanup
Rearrange the code little bit so that it is easier to read
Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@gmail.com>
---
contrib/gitview/gitview | 50 +++++++++++++++++++++--------------------------
1 files changed, 22 insertions(+), 28 deletions(-)
1dd075c97bae172d0c1b6f31897fec962217aab2
diff --git a/contrib/gitview/gitview b/contrib/gitview/gitview
index 02e2445..2cde71e 100755
--- a/contrib/gitview/gitview
+++ b/contrib/gitview/gitview
@@ -870,21 +870,22 @@ class GitView:
# Reset nodepostion
if (last_nodepos > 5):
- last_nodepos = 0
+ last_nodepos = -1
# Add the incomplete lines of the last cell in this
try:
colour = self.colours[commit.commit_sha1]
except KeyError:
- last_colour +=1
- self.colours[commit.commit_sha1] = last_colour
- colour = last_colour
+ self.colours[commit.commit_sha1] = last_colour+1
+ last_colour = self.colours[commit.commit_sha1]
+ colour = self.colours[commit.commit_sha1]
+
try:
node_pos = self.nodepos[commit.commit_sha1]
except KeyError:
- last_nodepos +=1
- self.nodepos[commit.commit_sha1] = last_nodepos
- node_pos = last_nodepos
+ self.nodepos[commit.commit_sha1] = last_nodepos+1
+ last_nodepos = self.nodepos[commit.commit_sha1]
+ node_pos = self.nodepos[commit.commit_sha1]
#The first parent always continue on the same line
try:
@@ -895,32 +896,25 @@ class GitView:
self.nodepos[commit.parent_sha1[0]] = node_pos
for sha1 in self.incomplete_line.keys():
- if ( sha1 != commit.commit_sha1):
+ if (sha1 != commit.commit_sha1):
self.draw_incomplete_line(sha1, node_pos,
out_line, in_line, index)
else:
del self.incomplete_line[sha1]
- in_line.append((node_pos, self.nodepos[commit.parent_sha1[0]],
- self.colours[commit.parent_sha1[0]]))
-
- self.add_incomplete_line(commit.parent_sha1[0], index+1)
-
- if (len(commit.parent_sha1) > 1):
- for parent_id in commit.parent_sha1[1:]:
- try:
- tmp_node_pos = self.nodepos[parent_id]
- except KeyError:
- last_colour += 1;
- self.colours[parent_id] = last_colour
- last_nodepos +=1
- self.nodepos[parent_id] = last_nodepos
-
- in_line.append((node_pos, self.nodepos[parent_id],
- self.colours[parent_id]))
- self.add_incomplete_line(parent_id, index+1)
-
+ for parent_id in commit.parent_sha1:
+ try:
+ tmp_node_pos = self.nodepos[parent_id]
+ except KeyError:
+ self.colours[parent_id] = last_colour+1
+ last_colour = self.colours[parent_id]
+ self.nodepos[parent_id] = last_nodepos+1
+ last_nodepos = self.nodepos[parent_id]
+
+ in_line.append((node_pos, self.nodepos[parent_id],
+ self.colours[parent_id]))
+ self.add_incomplete_line(parent_id)
try:
branch_tag = self.bt_sha1[commit.commit_sha1]
@@ -935,7 +929,7 @@ class GitView:
return (in_line, last_colour, last_nodepos)
- def add_incomplete_line(self, sha1, index):
+ def add_incomplete_line(self, sha1):
try:
self.incomplete_line[sha1].append(self.nodepos[sha1])
except KeyError:
--
1.2.3.g2cf3-dirty
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox