* Re: file rename causes history to disappear
From: Junio C Hamano @ 2006-09-07 0:52 UTC (permalink / raw)
To: Randal L. Schwartz; +Cc: Linus Torvalds, git
In-Reply-To: <86bqpsvfd3.fsf@blue.stonehenge.com>
merlyn@stonehenge.com (Randal L. Schwartz) writes:
>>>>>> "Junio" == Junio C Hamano <junkio@cox.net> writes:
>
> Junio> The only people who will get burnt by this change are the ones
> Junio> with metacharacters in their pathnames, so it is relative safe
> Junio> change.
>
> But does that mean you'll provide the equivalent to "fgrep" for "grep",
> as in a switch that turns this off, or a seperate command?
>
> I can think of times when I might be trying to track a file with a square
> bracket in the name.
If your path is "foo.c[1]" then "foo.c[1]" as fnmatch() pattern
would not obviously match it, which is sad.
However, we do try to match the path literally before falling
back to fnmatch() so in practice I do not think it is so bad.
$ git ls-files -s ;# everybody has "hello world".
100644 3b18e512dba79e4c8300dd08aeb37f8e728b8dad 0 foo.c
100644 3b18e512dba79e4c8300dd08aeb37f8e728b8dad 0 foo/bar[1]/baz/boa.c
100644 3b18e512dba79e4c8300dd08aeb37f8e728b8dad 0 foo/bar[2].c
$ git grep hello -- 'foo/bar[1]'
foo/bar[1]/baz/boa.c:hello world
$ git grep hello -- 'foo/bar[[]*[]]*'
foo/bar[1]/baz/boa.c:hello world
foo/bar[2].c:hello world
$ git grep hello -- 'fo*'
foo.c:hello world
foo/bar[1]/baz/boa.c:hello world
foo/bar[2].c:hello world
$ exit
^ permalink raw reply
* Re: A look at some alternative PACK file encodings
From: Linus Torvalds @ 2006-09-07 0:45 UTC (permalink / raw)
To: A Large Angry SCM; +Cc: Jon Smirl, git
In-Reply-To: <44FF6586.8080206@gmail.com>
On Wed, 6 Sep 2006, A Large Angry SCM wrote:
>
> > Btw, that "keeping the ordering it had" part I'm not convinced we actually
> > enforce. That would depend on the sort algorithm used by "qsort()", I
> > think. So there might be room for improvement there in order to keep
> > things in recency order.
>
> qsort() is not stable.
Well, quicksort isn't, but a lot of systems don't actually use quicksort
to implement qsort() - despite the name.
I think, for example, that glibc uses a merge-sort when there are lots of
entries (to avoid any potential bad behaviour with quicksort), and that
should be stable. But I didn't actually check..
In fact, I didn't even think of this issue originally, but it might
actually be worthwhile doing our own sort, exactly because we'll otherwise
have different systems generating different pack-files due to issues like
this.
So even if we don't actually _care_ whether the sorting is stable or not,
we might well care that we always get the same results, regardless of what
C library we have.
> > Is there any way to get zlib to just generate a suggested dictionary from
> > a given set of input?
>
> The docs suggest "no".
Oh, well.
Linus
^ permalink raw reply
* Re: [PATCH 2/2] Teach rev-list an option to read revs from the standard input.
From: Junio C Hamano @ 2006-09-07 0:44 UTC (permalink / raw)
To: Andy Whitcroft; +Cc: git
In-Reply-To: <44FEBFD6.10709@shadowen.org>
Andy Whitcroft <apw@shadowen.org> writes:
> Ok, I've tested this with an updated version of my patch to make
> send-pack use this (which I'll send out following this message) and it
> seems to work pretty well.
Eh, I have your earlier one placed on "pu" already. I do not
mind replacing it with the new one but (1) have you checked what
is in "pu"? (2) how different is this new one compared to it?
^ permalink raw reply
* Re: A look at some alternative PACK file encodings
From: Nicolas Pitre @ 2006-09-07 0:40 UTC (permalink / raw)
To: Jon Smirl; +Cc: gitzilla, git
In-Reply-To: <9e4733910609061623k73086dbey4a600ecf2852c024@mail.gmail.com>
On Wed, 6 Sep 2006, Jon Smirl wrote:
> Shawn is doing some prototype work on true dictionary based
> compression. I don't know how far along he is but it has potential for
> taking 30% off the Mozilla pack.
BTW I'm half-way done with support for deltas which base object is
referenced with an offset in the pack instead of a hash. It is quite
messy though since it touches pretty core code all over the place when
it comes to fetching objects out of a pack.
Nicolas
^ permalink raw reply
* Re: A look at some alternative PACK file encodings
From: Nicolas Pitre @ 2006-09-07 0:37 UTC (permalink / raw)
To: Linus Torvalds; +Cc: A Large Angry SCM, Jon Smirl, git
In-Reply-To: <Pine.LNX.4.64.0609061651500.27779@g5.osdl.org>
On Wed, 6 Sep 2006, Linus Torvalds wrote:
^ permalink raw reply
* Re: [PATCH 1/7] gitweb: Make pickaxe search a feature
From: Junio C Hamano @ 2006-09-07 0:37 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <1157548091229-git-send-email-jnareb@gmail.com>
Jakub Narebski <jnareb@gmail.com> writes:
> As pickaxe search (selected using undocumented 'pickaxe:' operator in
> search query) is resource consuming, allow to turn it on/off using
> feature meachanism.
>
> Signed-off-by: Jakub Narebski <jnareb@gmail.com>
I do not have a problem against making it configurable.
> + 'pickaxe' => {
> + 'sub' => \&feature_pickaxe,
> + 'override' => 0,
> + 'default' => [0]},
> );
The patch suggests that it is turned off by default right now; I
have not checked it myself, but is that the case?
^ permalink raw reply
* Re: A look at some alternative PACK file encodings
From: A Large Angry SCM @ 2006-09-07 0:19 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Jon Smirl, git
In-Reply-To: <Pine.LNX.4.64.0609061651500.27779@g5.osdl.org>
Linus Torvalds wrote:
>
> On Wed, 6 Sep 2006, A Large Angry SCM wrote:
>
>> Jon Smirl wrote:
>>> On 9/6/06, A Large Angry SCM <gitzilla@gmail.com> wrote:
>>>> TREE objects do not delta or deflate well.
>>> I can understand why they don't deflate, the path names are pretty
>>> much unique and the sha1s are incompressible. By why don't they delta
>>> well? Does sorting them by size mess up the delta process?
>> My guess would be the TREEs would only delta well against other TREE
>> versions for the same path.
>
> That's what you'd normally have in a real project, though. I wonder if
> your "pack mashup" lost the normal behaviour: we very much sort trees
> together normally, thanks to the "sort-by-filename, then by size"
> behaviour that git-pack-objects should have (for trees, the size normally
> shouldn't change, so the sorting should basically boil down to "sort the
> same directory together, keeping the ordering it had from git-rev-list").
The mashup is just all the projects in a single repository with a bushy
refs tree so I can view the updates in a single gitk window.
The sorting by name, then by path may be breaking the object version
relationship for wide graphs.
> Btw, that "keeping the ordering it had" part I'm not convinced we actually
> enforce. That would depend on the sort algorithm used by "qsort()", I
> think. So there might be room for improvement there in order to keep
> things in recency order.
qsort() is not stable.
>> Just looking at the structures in non-BLOBS, I see a lot of potential
>> for the use of a set dictionaries when deflating TREEs and another set
>> of dictionaries when deflating COMMITs and TAGs. The low hanging fruit
>> is to create dictionaries of the most referenced IDs across all TREE or
>> COMMIT/TAG objects.
>
> Is there any way to get zlib to just generate a suggested dictionary from
> a given set of input?
The docs suggest "no".
^ permalink raw reply
* Re: A look at some alternative PACK file encodings
From: David Lang @ 2006-09-07 0:06 UTC (permalink / raw)
To: Jon Smirl; +Cc: Linus Torvalds, A Large Angry SCM, git
In-Reply-To: <9e4733910609061710x4fb48d86o58b9c5ec8e527135@mail.gmail.com>
On Wed, 6 Sep 2006, Jon Smirl wrote:
> On 9/6/06, Linus Torvalds <torvalds@osdl.org> wrote:
>> Is there any way to get zlib to just generate a suggested dictionary from
>> a given set of input?
>
> No, I asked the author. Apparently it is a hard problem, there have
> been research papers written about it.
>
> Shawn has a Perl script that makes a guess at a dictionary. That
> scripts gets 4-7% improvement. The number one thing that ended up in
> the Mozilla dictionary was the five different license versions that
> had each been copied into 50,000 files over time.
for the mozilla project it may make sense to feed all these license files from
all over as one string to git, as an exception to your normal process of going
file by file. if you can do this then the delta functionality should reduce
these files to practicaly nothing.
David Lang
^ permalink raw reply
* Re: A look at some alternative PACK file encodings
From: Jon Smirl @ 2006-09-07 0:10 UTC (permalink / raw)
To: Linus Torvalds; +Cc: A Large Angry SCM, git
In-Reply-To: <Pine.LNX.4.64.0609061651500.27779@g5.osdl.org>
On 9/6/06, Linus Torvalds <torvalds@osdl.org> wrote:
> Is there any way to get zlib to just generate a suggested dictionary from
> a given set of input?
No, I asked the author. Apparently it is a hard problem, there have
been research papers written about it.
Shawn has a Perl script that makes a guess at a dictionary. That
scripts gets 4-7% improvement. The number one thing that ended up in
the Mozilla dictionary was the five different license versions that
had each been copied into 50,000 files over time.
I think what happens in practice is that when people get to the point
of trying to optimize zlib dictionaries they switch compression
methods.
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: A look at some alternative PACK file encodings
From: Jon Smirl @ 2006-09-07 0:04 UTC (permalink / raw)
To: gitzilla; +Cc: git
In-Reply-To: <44FF5C27.2040300@gmail.com>
On 9/6/06, A Large Angry SCM <gitzilla@gmail.com> wrote:
> Jon Smirl wrote:
> > On 9/6/06, A Large Angry SCM <gitzilla@gmail.com> wrote:
> >> TREE objects do not delta or deflate well.
> >
> > I can understand why they don't deflate, the path names are pretty
> > much unique and the sha1s are incompressible. By why don't they delta
> > well? Does sorting them by size mess up the delta process?
>
> My guess would be the TREEs would only delta well against other TREE
> versions for the same path.
I would have thought that the TREE delta code would have already taken
this into account. Is there enough info around to match up all of the
TREEs for a specific path into a delta chain?
The repack code could build a model of the tree as it is repacking,
that is what fast-import does. If you have a model of the tree then
when you change a TREE node you track the last sha1 that corresponded
to that directory path. Now you know what to diff to.
> > Shawn is doing some prototype work on true dictionary based
> > compression. I don't know how far along he is but it has potential for
> > taking 30% off the Mozilla pack.
>
> Just looking at the structures in non-BLOBS, I see a lot of potential
> for the use of a set dictionaries when deflating TREEs and another set
> of dictionaries when deflating COMMITs and TAGs. The low hanging fruit
> is to create dictionaries of the most referenced IDs across all TREE or
> COMMIT/TAG objects.
Doing this will get 4-7% according to our small tests, to get more you
need a different type of compression algorithm. For true dictionary
based compression you parse all of the input documents into words. The
words are put into a single giant huffman table. Then the huffman
codes for the words are concatenated together to form the compressed
document. This is one of the highest possible compression methods
since it takes into account you are compressing something that is
human readable. gzip is sort of like this but the dictionary is
limited to 32KB.
Search engines use true dictionary compression since it is easy to add
on search indices to the compressed data. For a search index you make
a bit vector that corresponds to each word in the dictionary and then
set a bit if that document number contains the word. The vectors and
dictionary are stored compressed. There are algorithms for doing
and/or on the compressed vectors. The and/or stage lets you identify
likely hits, you retrieve those and then do an exact match on the
search terms to be sure.
--
Jon Smirl
jonsmirl@gmail.com
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: A look at some alternative PACK file encodings
From: Linus Torvalds @ 2006-09-06 23:56 UTC (permalink / raw)
To: A Large Angry SCM; +Cc: Jon Smirl, git
In-Reply-To: <44FF5C27.2040300@gmail.com>
On Wed, 6 Sep 2006, A Large Angry SCM wrote:
> Jon Smirl wrote:
> > On 9/6/06, A Large Angry SCM <gitzilla@gmail.com> wrote:
> >> TREE objects do not delta or deflate well.
> >
> > I can understand why they don't deflate, the path names are pretty
> > much unique and the sha1s are incompressible. By why don't they delta
> > well? Does sorting them by size mess up the delta process?
>
> My guess would be the TREEs would only delta well against other TREE
> versions for the same path.
That's what you'd normally have in a real project, though. I wonder if
your "pack mashup" lost the normal behaviour: we very much sort trees
together normally, thanks to the "sort-by-filename, then by size"
behaviour that git-pack-objects should have (for trees, the size normally
shouldn't change, so the sorting should basically boil down to "sort the
same directory together, keeping the ordering it had from git-rev-list").
Btw, that "keeping the ordering it had" part I'm not convinced we actually
enforce. That would depend on the sort algorithm used by "qsort()", I
think. So there might be room for improvement there in order to keep
things in recency order.
> Just looking at the structures in non-BLOBS, I see a lot of potential
> for the use of a set dictionaries when deflating TREEs and another set
> of dictionaries when deflating COMMITs and TAGs. The low hanging fruit
> is to create dictionaries of the most referenced IDs across all TREE or
> COMMIT/TAG objects.
Is there any way to get zlib to just generate a suggested dictionary from
a given set of input?
Linus
^ permalink raw reply
* Re: file rename causes history to disappear
From: Junio C Hamano @ 2006-09-06 23:54 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git, Jakub Narebski
In-Reply-To: <Pine.LNX.4.64.0609061205100.27779@g5.osdl.org>
Linus Torvalds <torvalds@osdl.org> writes:
> On Wed, 6 Sep 2006, Jakub Narebski wrote:
>>
>> But --follow=<filename> with <pathspec> can be useful, e.g. when <pathspec>
>> is a directory (or, perhaps in the future, glob), which would mean "follow
>> the contents indicated in starting hash by <filename>, and stop following
>> when it falls out outside given <pathspec>, in our case given directory".
>
> Yes, that would indeed make sense. The pathspec ends up being kept as a
> "limiter", and basically tells you what the "context" for following is
> allowed to be.
>
> Color me convinced.
Likewise.
^ permalink raw reply
* Re: A look at some alternative PACK file encodings
From: A Large Angry SCM @ 2006-09-06 23:39 UTC (permalink / raw)
To: Jon Smirl; +Cc: git
In-Reply-To: <9e4733910609061623k73086dbey4a600ecf2852c024@mail.gmail.com>
Jon Smirl wrote:
> On 9/6/06, A Large Angry SCM <gitzilla@gmail.com> wrote:
>> TREE objects do not delta or deflate well.
>
> I can understand why they don't deflate, the path names are pretty
> much unique and the sha1s are incompressible. By why don't they delta
> well? Does sorting them by size mess up the delta process?
My guess would be the TREEs would only delta well against other TREE
versions for the same path.
> Shawn is doing some prototype work on true dictionary based
> compression. I don't know how far along he is but it has potential for
> taking 30% off the Mozilla pack.
Just looking at the structures in non-BLOBS, I see a lot of potential
for the use of a set dictionaries when deflating TREEs and another set
of dictionaries when deflating COMMITs and TAGs. The low hanging fruit
is to create dictionaries of the most referenced IDs across all TREE or
COMMIT/TAG objects.
^ permalink raw reply
* Re: A look at some alternative PACK file encodings
From: Jon Smirl @ 2006-09-06 23:23 UTC (permalink / raw)
To: gitzilla; +Cc: git
In-Reply-To: <44FF41F4.1090906@gmail.com>
On 9/6/06, A Large Angry SCM <gitzilla@gmail.com> wrote:
> TREE objects do not delta or deflate well.
I can understand why they don't deflate, the path names are pretty
much unique and the sha1s are incompressible. By why don't they delta
well? Does sorting them by size mess up the delta process?
Shawn is doing some prototype work on true dictionary based
compression. I don't know how far along he is but it has potential for
taking 30% off the Mozilla pack.
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: [PATCH 0/7] gitweb: Trying to improve history view speed
From: Junio C Hamano @ 2006-09-06 22:01 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <200609061504.40725.jnareb@gmail.com>
Jakub Narebski <jnareb@gmail.com> writes:
> gitweb/gitweb.perl | 180 +++++++++++++++++++++++++++++++++++++++++-----------
> 1 files changed, 141 insertions(+), 39 deletions(-)-
>
> P.S. Is putting diffstat in such a series of patches actually usefull?
Very much -- it shows me that you are not damaging the core but
only touching gitweb.
^ permalink raw reply
* Re: [PATCH 8/8] gitweb: Remove --parents from call to git-rev-list in parse_rev_list
From: Jakub Narebski @ 2006-09-06 21:53 UTC (permalink / raw)
To: git
In-Reply-To: <11575761821830-git-send-email-jnareb@gmail.com>
Jakub Narebski wrote:
> Benchmarks (7 means patch before, 8 means this patch):
> # 1:gitweb/new~n 2:%e 3:%U 4:%s 5:ab-n10_cgi_time[ms] 6:[+/-sd] 7:ab-n10_perl_time[ms] 8:[+/-sd]
> 7 2.59 1.53 0 2621.073 234.2 2742.230 96.6
> 8 2.89 1.80 0 3081.722 246.6 3306.196 367.2
> 8 2.95 1.81 0 2952.253 155.9 3175.441 128.0
Corrected benchmark (on the same state of repository) are
7 2.67 1.60 0 2694.654 275.2 2866.028 168.4
8 2.84 1.86 0 2892.864 263.0 3135.065 68.1
where the benchmarking commands are
$ /usr/bin/time -f "%e %U %s" \
GATEWAY_INTERFACE="CGI/1.1" HTTP_ACCEPT="*/*" REQUEST_METHOD="GET" \
QUERY_STRING="p=git.git;a=history;f=gitweb/gitweb.perl" \
GITWEB_CONFIG="~/git/gitweb/gitweb_config.perl \
perl -- ~/git/gitweb/gitweb.perl
$ ab -n 10 "http://localhost/cgi-bin/gitweb/gitweb.cgi?p=git.git;a=history;f=gitweb/gitweb.perl"
$ ab -n 10 "http://localhost/perl/gitweb/gitweb.cgi?p=git.git;a=history;f=gitweb/gitweb.perl"
and we get version 7 using
$ git checkout HEAD^ -- gitweb/gitweb.perl
and updating gitweb.cgi in correct place.
So actually removing '--parents' option did make gitweb _slower_ (sic!),
if only slightly.
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: [PATCH 8/8] gitweb: Remove --parents from call to git-rev-list in parse_rev_list
From: Junio C Hamano @ 2006-09-06 21:51 UTC (permalink / raw)
Cc: git
In-Reply-To: <edndud$csb$1@sea.gmane.org>
Jakub Narebski <jnareb@gmail.com> writes:
> Linus Torvalds wrote:
>
>> But I guess for gitweb, you do want to use --full-history in this case ;(
>
> It is now easy with patch 3/7 "Use @hist_opts as git-rev-list parameters in
> git_history" to remove '--full-history' from git-rev-list parameters in
> git_history subroutine.
Well, generating not-so-useful output fast is, eh, not-so-useful
anyway, so being able to add or not add to the set of options on
a whim is not so much of value. The real value lies in deciding
if it is needed (and I think --full-history is needed for gitweb)
and if so coming up with a way to get the same result with
cheaper operations.
^ permalink raw reply
* A look at some alternative PACK file encodings
From: A Large Angry SCM @ 2006-09-06 21:47 UTC (permalink / raw)
To: git
Since I'm currently between jobs and extra time on my hands until I find
a new one, I've been experimenting with some alternative encodings for
PACK file contents.
The tables below show packing statistics for a number of actual pack
files using difference pack file encodings. All of the encodings use
object names to find the base object of a DELTA object.
Encodings:
Pack_2 - Pack version 2 (Git current), 32 bit
Pack_3 - Pack version 3 (Git future), 32 bit
Gitzilla_1 - Experimental >32 bit, described below
Gitzilla_2 - Experimental >32 bit, described below
Gitzilla_1: Per object header consisting of 2 fields: the Git object
type of the object with a flag indicating if it's delta encoded, big
endian base 128 encoding of the object length. If the object is not
delta encoded, the object header is followed by the deflated object
contents. If the object is delta encoded, the object header is followed
by the 20 byte ID of the base object, and the deflated edit list (see
below). This encoding allows for the possibility of using some of the
bits in the object type field to select one of up to 15 dictionaries.
Gitzilla_2: Per object header consisting of a single field combining the
object type and length; similar to that used in current packs but big
endian encoded. The type could be encoded either as object type with
delta flag (no room for additional object types), or the same as what
version 2 packs (room for additional object types but need to walk the
delta chain to determine type of delta encode objects). If the object is
delta encoded, the object header is followed by the 20 byte ID of the
base object, and the deflated edit list (see below).
Notice that the length encoded in the per object header for delta encode
objects is that of the decoded object, not the length of the inflated
edit list.
The edit lists used in the Gitzilla encodings consist of a sequence of
one or more hunks. Each hunk specifies either a part of the base object
to copy to the current position of the result object, or a length of
(immediate) data to copy to the current position of the result object.
Each hunk type is encode as 2 fields: the first is big endian base 128
encoded length of data to copy with bit 6 of the first byte indicating
where to copy from. The second field is the big endian base 128 encoded
offset in the base object of the copy source, or the (immediate) data to
copy.
Observations:
Most of the savings are from not including the base and result object
lengths in the deflated edit lists.
Other minor savings from joining adjacent edit hunks of the same type,
where possible.
The endian of the base 128 encoding is big because it requires less work
to _decode_ than little.
TREE objects do not delta or deflate well.
Dictionaries will need to be tailored to different object types, and
possibly to other object subsets.
The ability to get the type and length of a delta'd object without
needing to walk the delta chain is a good thing.
The use of _relative_ offsets to specify base objects would save a lot
of space.
The data:
All of the PACK files except bd5109... were rsync'd from the Git project
on kernel.org, at some point. PACK bd5109... is a mash-up of Linus'
kernel and sparse projects, Thomas' kernel history, the stable 2.6.x.y
kernel projects, Git, and gitk; all from kernel.org.
The columns are:
encoding name
pack file size
size of pack file non-object overhead
sum of packed object sizes
byte difference in sum of packed object sizes and that of
version 2 packs
average byte difference per object
average byte difference per DELTA'd object
pack-1abe018fa97f8a257a4f3a072000abf99eaa925e.pack
BLOB : 237
COMMIT: 504
DELTA : 1108
TAG : 6
TREE : 64
====== ==========
Total : 1919
Gitzilla_1: 1202917 32 1202885 -6797 -3.5419 -6.1345
Gitzilla_2: 1202011 32 1201979 -7703 -4.0141 -6.9522
Pack_2 : 1209714 32 1209682 0 0.0000 0.0000
Pack_3 : 1209686 32 1209654 -28 -0.0146 -0.0253
pack-3a080dc749b452cc847d19d91981ddfcff629750.pack
BLOB : 1020
COMMIT: 5300
DELTA : 15013
TAG : 80
TREE : 564
====== ==========
Total : 21977
Gitzilla_1: 6748060 32 6748028 -95768 -4.3576 -6.3790
Gitzilla_2: 6737992 32 6737960 -105836 -4.8158 -7.0496
Pack_2 : 6843828 32 6843796 0 0.0000 0.0000
Pack_3 : 6843747 32 6843715 -81 -0.0037 -0.0054
pack-3e6247e4edf3c006a037041eef28bbca8051b006.pack
BLOB : 1062
COMMIT: 5545
DELTA : 15565
TAG : 81
TREE : 703
====== ==========
Total : 22956
Gitzilla_1: 6724940 32 6724908 -97643 -4.2535 -6.2732
Gitzilla_2: 6714303 32 6714271 -108280 -4.7168 -6.9566
Pack_2 : 6822583 32 6822551 0 0.0000 0.0000
Pack_3 : 6822458 32 6822426 -125 -0.0054 -0.0080
pack-6bdf691abdae3590f6a807d15ebe8ed523ef1e12.pack
BLOB : 33
COMMIT: 36
DELTA : 64
TAG : 0
TREE : 4
====== ==========
Total : 137
Gitzilla_1: 136041 32 136009 -273 -1.9927 -4.2656
Gitzilla_2: 135991 32 135959 -323 -2.3577 -5.0469
Pack_2 : 136314 32 136282 0 0.0000 0.0000
Pack_3 : 136314 32 136282 0 0.0000 0.0000
pack-740c99c0be6734adbd130737dec2608dc4682761.pack
BLOB : 181
COMMIT: 173
DELTA : 383
TAG : 5
TREE : 24
====== ==========
Total : 766
Gitzilla_1: 586580 32 586548 -1799 -2.3486 -4.6971
Gitzilla_2: 586264 32 586232 -2115 -2.7611 -5.5222
Pack_2 : 588379 32 588347 0 0.0000 0.0000
Pack_3 : 588370 32 588338 -9 -0.0117 -0.0235
pack-793a9e93286d6c656941977d2e5b49e28566edcd.pack
BLOB : 1158
COMMIT: 6512
DELTA : 18715
TAG : 90
TREE : 820
====== ==========
Total : 27295
Gitzilla_1: 7957552 32 7957520 -120209 -4.4041 -6.4231
Gitzilla_2: 7944784 32 7944752 -132977 -4.8718 -7.1054
Pack_2 : 8077761 32 8077729 0 0.0000 0.0000
Pack_3 : 8077566 32 8077534 -195 -0.0071 -0.0104
pack-92b2a535e63e4a97eba2a13f60c3da922372f03a.pack
BLOB : 1149
COMMIT: 5969
DELTA : 16930
TAG : 84
TREE : 750
====== ==========
Total : 24882
Gitzilla_1: 7243503 32 7243471 -108475 -4.3596 -6.4073
Gitzilla_2: 7231901 32 7231869 -120077 -4.8259 -7.0926
Pack_2 : 7351978 32 7351946 0 0.0000 0.0000
Pack_3 : 7351841 32 7351809 -137 -0.0055 -0.0081
pack-9f1f94e6f50261b5d99c04349bfa3d315cfb4118.pack
BLOB : 1190
COMMIT: 6244
DELTA : 17881
TAG : 87
TREE : 810
====== ==========
Total : 26212
Gitzilla_1: 7920476 32 7920444 -115214 -4.3955 -6.4434
Gitzilla_2: 7908256 32 7908224 -127434 -4.8617 -7.1268
Pack_2 : 8035690 32 8035658 0 0.0000 0.0000
Pack_3 : 8035529 32 8035497 -161 -0.0061 -0.0090
pack-a875e0c11c739866b000dc9b34d1b0daead84a00.pack
BLOB : 20
COMMIT: 9
DELTA : 15
TAG : 1
TREE : 3
====== ==========
Total : 48
Gitzilla_1: 71748 32 71716 -54 -1.1250 -3.6000
Gitzilla_2: 71728 32 71696 -74 -1.5417 -4.9333
Pack_2 : 71802 32 71770 0 0.0000 0.0000
Pack_3 : 71802 32 71770 0 0.0000 0.0000
pack-abbe9cdea12803663015a7e23b7ae61c64e56499.pack
BLOB : 259
COMMIT: 342
DELTA : 705
TAG : 13
TREE : 47
====== ==========
Total : 1366
Gitzilla_1: 1086337 32 1086305 -3878 -2.8389 -5.5007
Gitzilla_2: 1085773 32 1085741 -4442 -3.2518 -6.3007
Pack_2 : 1090215 32 1090183 0 0.0000 0.0000
Pack_3 : 1090215 32 1090183 0 0.0000 0.0000
pack-bd5109ab74dd67d5b0a42f26688a4fa0a579e9fa.pack
BLOB : 29212
COMMIT: 103241
DELTA : 702986
TAG : 182
TREE : 37102
====== ==========
Total : 872723
Gitzilla_1: 282786611 32 282786579 -3647429 -4.1794 -5.1885
Gitzilla_2: 282143551 32 282143519 -4290489 -4.9162 -6.1032
Pack_2 : 286434040 32 286434008 0 0.0000 0.0000
Pack_3 : 286404302 32 286404270 -29738 -0.0341 -0.0423
pack-d18e6cccf9e472ba05e108254c827cc7ac0fe2a9.pack
BLOB : 1564
COMMIT: 3118
DELTA : 8288
TAG : 51
TREE : 365
====== ==========
Total : 13386
Gitzilla_1: 5638236 32 5638204 -44212 -3.3029 -5.3345
Gitzilla_2: 5632126 32 5632094 -50322 -3.7593 -6.0717
Pack_2 : 5682448 32 5682416 0 0.0000 0.0000
Pack_3 : 5682423 32 5682391 -25 -0.0019 -0.0030
pack-d25faec39a55935d787ae66e4f4a69d3add16417.pack
BLOB : 1180
COMMIT: 6090
DELTA : 17458
TAG : 85
TREE : 761
====== ==========
Total : 25574
Gitzilla_1: 7646842 32 7646810 -112816 -4.4114 -6.4621
Gitzilla_2: 7634964 32 7634932 -124694 -4.8758 -7.1425
Pack_2 : 7759658 32 7759626 0 0.0000 0.0000
Pack_3 : 7759521 32 7759489 -137 -0.0054 -0.0078
pack-d453bb50fef8757a0d7f9fcc88429e9ce6f74077.pack
BLOB : 28
COMMIT: 35
DELTA : 46
TAG : 2
TREE : 6
====== ==========
Total : 117
Gitzilla_1: 177544 32 177512 -266 -2.2735 -5.7826
Gitzilla_2: 177478 32 177446 -332 -2.8376 -7.2174
Pack_2 : 177810 32 177778 0 0.0000 0.0000
Pack_3 : 177806 32 177774 -4 -0.0342 -0.0870
pack-d83305b8608cb685ecd025fb141bf9d1588ae9c5.pack
BLOB : 91
COMMIT: 108
DELTA : 202
TAG : 2
TREE : 26
====== ==========
Total : 429
Gitzilla_1: 473503 32 473471 -1112 -2.5921 -5.5050
Gitzilla_2: 473269 32 473237 -1346 -3.1375 -6.6634
Pack_2 : 474615 32 474583 0 0.0000 0.0000
Pack_3 : 474603 32 474571 -12 -0.0280 -0.0594
pack-ee22eefd6bad2b54358d52ad6221ecabdc79d6f5.pack
BLOB : 175
COMMIT: 210
DELTA : 541
TAG : 2
TREE : 38
====== ==========
Total : 966
Gitzilla_1: 662650 32 662618 -3230 -3.3437 -5.9704
Gitzilla_2: 662150 32 662118 -3730 -3.8613 -6.8946
Pack_2 : 665880 32 665848 0 0.0000 0.0000
Pack_3 : 665865 32 665833 -15 -0.0155 -0.0277
pack-f0c530478b9dd6e96a69482066ec66eb5f6edbbf.pack
BLOB : 1154
COMMIT: 5797
DELTA : 16300
TAG : 83
TREE : 739
====== ==========
Total : 24073
Gitzilla_1: 7212953 32 7212921 -102252 -4.2476 -6.2731
Gitzilla_2: 7201730 32 7201698 -113475 -4.7138 -6.9617
Pack_2 : 7315205 32 7315173 0 0.0000 0.0000
Pack_3 : 7315059 32 7315027 -146 -0.0061 -0.0090
^ permalink raw reply
* Re: [PATCH][RFC] Add git-archive-tree
From: Junio C Hamano @ 2006-09-06 21:47 UTC (permalink / raw)
To: Rene Scharfe; +Cc: git, Franck Bui-Huu
In-Reply-To: <44FF0DDE.7030700@lsrfire.ath.cx>
Rene Scharfe <rene.scharfe@lsrfire.ath.cx> writes:
> I then let the two chew away on the kernel repository. And as
> kcachegrind impressively shows, all we do with our trees and objects is
> dwarfed by inflate().
The diff output codepath has a logic that says "if the blob we
are dealing with has the same object name as the corresponding
blob in the index, and if the index entry is clean (i.e. it is
known that the file sitting in the working tree matches the
blob), then do not inflate() but use data from that file
instead". It was originally done that way because we used to
prepare temporary files out of blob and fed them to GNU diff,
but I think it is still kept that way. I wonder if doing the
same may be cheaper for archivers.
^ permalink raw reply
* Re: file rename causes history to disappear
From: Randal L. Schwartz @ 2006-09-06 21:45 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Linus Torvalds, git
In-Reply-To: <7vmz9c7pzm.fsf@assigned-by-dhcp.cox.net>
>>>>> "Junio" == Junio C Hamano <junkio@cox.net> writes:
Junio> The only people who will get burnt by this change are the ones
Junio> with metacharacters in their pathnames, so it is relative safe
Junio> change.
But does that mean you'll provide the equivalent to "fgrep" for "grep",
as in a switch that turns this off, or a seperate command?
I can think of times when I might be trying to track a file with a square
bracket in the name.
--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
^ permalink raw reply
* Re: [PATCH 1/2] Add git-archive
From: Junio C Hamano @ 2006-09-06 21:42 UTC (permalink / raw)
To: Franck; +Cc: git
In-Reply-To: <44FED12E.7010409@innova-card.com>
Franck Bui-Huu <vagabon.xyz@gmail.com> writes:
>>> +typedef int (*parse_extra_args_fn_t)(int argc,
>>> + const char **argv,
>>> + const char **reason);
>>> +
>>
>> I do not see a way for parse_extra to record the parameter it
>> successfully parsed, other than in a source-file-global, static
>> variable. Not a very nice design for a library, if we are
>> building one from scratch.
>
> Interesting, could you explain why static variables are not nice ?
Mostly taste and a little bit of re-entrancy worries.
> You might have missed my second patch:
>
> "[PATCH 2/2] Add git-upload-archive"
>
> Basically the server can also use 'reason' to report a failure
> description during NACK. I find it more useful than the simple
> "server sent EOF" error message.
That's a good intention, but we would also need to convey the
"server side found problem and died with these error() output"
anyway, so it would be covered either way (see how error()/die()
messages from git-upload-pack are given to git-fetch-pack over
the wire).
> 'remote' case is not a generic argument that can be passed to
> archiver backends. Remember, the archiver backends only do local
> operation. They do not know about remote protocol which is part
> of git-archive command. That's the reason why I think we shouldn't
> make this field part of arguments structure.
Ok. Passing that as a separate paramter would make sense.
>> After parse_archive_args finds the archiver specified with
>> --format=*, it can call its parse_extra to retrieve a suitable
>> struct that has struct archive_args embedded at the beginning,
>> and then set remote and prefix on the returned structure.
>
> One bad side is that we need to malloc this embedded structure.
Not at all, if you read the example I did you would notice that
I changed parse_extra for each backend to return this structure
allocated for that particular backend.
>>> +static int run_remote_archiver(struct archiver_struct *ar, int argc,
>>> + const char **argv)
>>> +{
>>> + char *url, buf[1024];
>>> + pid_t pid;
>>> + int fd[2];
>>> + int len, rv;
>>> +
>>> + sprintf(buf, "git-upload-%s", ar->name);
>>
>> Are you calling git-upload-{tar,zip,rar,...} here?
>
> yes. Actually git-upload-{tar,zip,...} commands are going to be
> removed, but git-daemon know them as a daemon service.
That would break "git-archive --remove=ssh://site/repo treeish"
wouldn't it?
^ permalink raw reply
* Re: [PATCH 8/8] gitweb: Remove --parents from call to git-rev-list in parse_rev_list
From: Jakub Narebski @ 2006-09-06 21:18 UTC (permalink / raw)
To: git
In-Reply-To: <Pine.LNX.4.64.0609061404490.27779@g5.osdl.org>
Linus Torvalds wrote:
> On Wed, 6 Sep 2006, Jakub Narebski wrote:
>>
>> Benchmarks (7 means patch before, 8 means this patch):
>
> Btw, you should possibly look at cold-cache numbers, and numbers for
> projects that aren't fully packed. They can often be _dramatically_
> different.
By the way I forgot that the case 8 is for the repository with 1 commit
more, although that shouldn't matter much for paginated output. Still, it
is one commit more unpacked. Benchmark for 7 was also for partially packed
repository.
> That said, the dramatic change would probably be if there were some way to
> avoid using "--full-history" (rather than "--parents", which doesn't add
> _that_ much overhead), since that "follow all parents" behaviour of
> full-history is usually what really makes a big deal.
>
> But I guess for gitweb, you do want to use --full-history in this case ;(
It is now easy with patch 3/7 "Use @hist_opts as git-rev-list parameters in
git_history" to remove '--full-history' from git-rev-list parameters in
git_history subroutine. Or add '--remove-empty' which matters only for the
last page of file/directory history output.
I had some simple benchmark that shown that the earlier version with
filtering via piping git-rev-list to git-diff-tree --stdin -- <filename>
was slightly faster than git-rev-list --full-history -- <filename>
(current version). If I remember correctly of course. And this version can
be easily extended to include renames (but not file to directory changes).
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: [PATCH 8/8] gitweb: Remove --parents from call to git-rev-list in parse_rev_list
From: Linus Torvalds @ 2006-09-06 21:08 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <11575761821830-git-send-email-jnareb@gmail.com>
On Wed, 6 Sep 2006, Jakub Narebski wrote:
>
> Benchmarks (7 means patch before, 8 means this patch):
Btw, you should possibly look at cold-cache numbers, and numbers for
projects that aren't fully packed. They can often be _dramatically_
different.
That said, the dramatic change would probably be if there were some way to
avoid using "--full-history" (rather than "--parents", which doesn't add
_that_ much overhead), since that "follow all parents" behaviour of
full-history is usually what really makes a big deal.
But I guess for gitweb, you do want to use --full-history in this case ;(
Linus
^ permalink raw reply
* [PATCH 8/8] gitweb: Remove --parents from call to git-rev-list in parse_rev_list
From: Jakub Narebski @ 2006-09-06 20:56 UTC (permalink / raw)
To: git; +Cc: Jakub Narebski
In-Reply-To: <11575480921132-git-send-email-jnareb@gmail.com>
This option is removed by default, because with --full-history option
(which is by default used in git_history) --parents means full history
with parenthood, which means that we get all the connecting merges
too. And since we asked for the _full_ history, that means EVERY
SINGLE MERGE. Even those that do not change given file (or
directory).
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
Benchmarks (7 means patch before, 8 means this patch):
# 1:gitweb/new~n 2:%e 3:%U 4:%s 5:ab-n10_cgi_time[ms] 6:[+/-sd] 7:ab-n10_perl_time[ms] 8:[+/-sd]
7 2.59 1.53 0 2621.073 234.2 2742.230 96.6
8 2.89 1.80 0 3081.722 246.6 3306.196 367.2
8 2.95 1.81 0 2952.253 155.9 3175.441 128.0
gitweb/gitweb.perl | 9 ++++++++-
1 files changed, 8 insertions(+), 1 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index edded74..68ddbe6 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -1131,6 +1131,13 @@ # to parse individual commits.
#
# parse_rev_list parameters are passed to git-rev-list, so they should
# include at least starting revision; just in case we default to HEAD
+#
+# if you want in parsed info to have full equivalent of first generating
+# list of interesting revisions, then calling parse_commit on each of revs,
+# i.e. if you want to have full parent info which includes grafts,
+# you have to include '--parents' in the parse_rev_list parameters;
+# it is excluded by default as it changes notion which revs are interesting
+# e.g. '--full-history' with '--parents' include EVERY SINGLE MERGE.
sub parse_rev_list {
my @rev_opts = @_;
my @revlist;
@@ -1138,7 +1145,7 @@ sub parse_rev_list {
@rev_opts = ("HEAD") unless @rev_opts;
local $/ = "\0";
- open my $fd, "-|", git_cmd(), "rev-list", "--header", "--parents", @rev_opts
+ open my $fd, "-|", git_cmd(), "rev-list", "--header", @rev_opts
or return \@revlist;
while (my $revinfo = <$fd>) {
--
1.4.2
^ permalink raw reply related
* Re: [PATCH 1/2] Add git-archive
From: Jakub Narebski @ 2006-09-06 20:29 UTC (permalink / raw)
To: git
In-Reply-To: <44FF2C37.2010400@lsrfire.ath.cx>
Rene Scharfe wrote:
> IMHO should work like in the following example, and the code above
> cuts off the Documentation part:
>
> $ cd Documentation
> $ git-archive --format=tar --prefix=v1.0/ HEAD howto | tar tf -
> v1.0/howto/
> v1.0/howto/isolate-bugs-with-bisect.txt
> ...
>
> I agree that simple subtree matching would be enough, at least for
> now.
What about
$ git-archive --format=tar --prefix=v1.0/ HEAD:Documentation/howto
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox