What's in git.git

What's in git.git

[Junio C Hamano]

Executive Summary.

[maint]

  I might do a 1.4.3.5 with the accumulated stuff, but the
  stablization cycle for v1.4.4 has started tonight, so it may
  not be worth the effort, unless something more pressing comes
  along.

[master]

  Three topics that have been cooking in 'next' have been
  merged, I've tagged the tip as v1.4.4-rc1.

   - Nico and Shawn's keep-pack work;

   - Loosening of bogusly overstrict 'a working tree file will
     be overwritten by the merge' check;

   - git-pickaxe.

[next]

  This now is almost empty, and I'd like to keep it that way
  until v1.4.5 final.  IOW, I'd be happier if people sent in
  only obviously correct fixes to 'master' than seeing the next
  greatest feature ;-).

  By the way, do people mind if I start to rewind and rebase
  'next' after every feature release (i.e. tagged release is
  made after 'master')?  I do not feel a strong need for it, and
  'git log --no-merges master..next' will show emptiness
  eventually, but being able to restart from clean slate after a
  release would be somewhat nice.

[pu]

  Johannes's shallow clone work now should rebase cleanly on top
  of 'master' although I haven't done so yet.  As he said
  himself the series is waiting for people who have needs for
  such a feature to raise hands.

  The part #2 of git-diff/git-apply change has a slight backward
  compatibility issue, and until everybody who is affected is
  upgraded to v1.4.3 (which has already prepared us for this
  change) we cannot push it out to 'master'.  It adjusts the
  diff output header for files with SP in their names to what
  GNU patch accepts.

----------------------------------------------------------------

* The 'maint' branch has these fixes since the last announcement.

   Alex Riesen (1):
      merge-recursive implicitely depends on trust_executable_bit

   Andy Parkins (2):
      Minor grammar fixes for git-diff-index.txt
      git-clone documentation didn't mention --origin as equivalent of -o

   Jakub Narebski (1):
      Documentation: Transplanting branch with git-rebase --onto

   Jeff King (1):
      Fix git-runstatus for repositories containing a file named HEAD

   Johannes Schindelin (1):
      link_temp_to_file: call adjust_shared_perm() only when we created the directory

   Junio C Hamano (2):
      apply: handle "traditional" creation/deletion diff correctly.
      adjust_shared_perm: chmod() only when needed.

   Shawn O. Pearce (3):
      Use ULONG_MAX rather than implicit cast of -1.
      Remove SIMPLE_PROGRAMS and make git-daemon a normal program.
      Remove unsupported C99 style struct initializers in git-archive.

   Tero Roponen (1):
      remove an unneeded test

* The 'master' branch has these since the last announcement.

   Alex Riesen (1):
      merge-recursive implicitely depends on trust_executable_bit

   Alexandre Julliard (5):
      pack-refs: Store the full name of the ref even when packing only tags.
      git.el: Added functions for moving to the next/prev unmerged file.
      git.el: Added a function to open the current file in another window.
      git.el: Move point after the log message header when entering log-edit mode.
      git.el: Include MERGE_MSG in the log-edit buffer even when not committing a merge.

   Andy Parkins (3):
      Remove uneccessarily similar printf() from print_ref_list() in builtin-branch
      Minor grammar fixes for git-diff-index.txt
      git-clone documentation didn't mention --origin as equivalent of -o

   Aneesh Kumar K.V (1):
      gitweb: Remove extra "/" in path names for git_get_project_list

   Eric Wong (2):
      git-svn: avoid printing filenames of files we're not tracking
      git-svn: don't die on rebuild when --upgrade is specified

   Jakub Narebski (4):
      gitweb: Use git-for-each-ref to generate list of heads and/or tags
      gitweb: Output also empty patches in "commitdiff" view
      gitweb: Better support for non-CSS aware web browsers
      Documentation: Transplanting branch with git-rebase --onto

   Jeff King (2):
      git-pickaxe: work properly in a subdirectory.
      Fix git-runstatus for repositories containing a file named HEAD

   Johannes Schindelin (1):
      link_temp_to_file: call adjust_shared_perm() only when we created the directory

   Junio C Hamano (40):
      git-pickaxe: blame rewritten.
      git-pickaxe -M: blame line movements within a file.
      git-pickaxe -C: blame cut-and-pasted lines.
      git-pickaxe: pagenate output by default.
      git-pickaxe: fix nth_line()
      git-pickaxe: improve "best match" heuristics
      git-pickaxe: introduce heuristics to avoid "trivial" chunks
      git-pickaxe: do not keep commit buffer.
      git-pickaxe: do not confuse two origins that are the same.
      git-pickaxe: get rid of wasteful find_origin().
      git-pickaxe: swap comparison loop used for -C
      merge: loosen overcautious "working file will be lost" check.
      merge-recursive: use abbreviated commit object name.
      merge-recursive: make a few functions static.
      merge-recursive: adjust to loosened "working file clobbered" check
      t6022: ignoring untracked files by merge-recursive when they do not matter
      send-pack --keep: do not explode into loose objects on the receiving end.
      git-pickaxe: WIP to refcount origin structure.
      git-pickaxe: allow -Ln,m as well as -L n,m
      git-pickaxe: refcount origin correctly in find_copy_in_parent()
      git-pickaxe: tighten sanity checks.
      Revert "send-pack --keep: do not explode into loose objects on the receiving end."
      git-pickaxe: split find_origin() into find_rename() and find_origin().
      git-pickaxe: cache one already found path per commit.
      Introduce a new revision set operator <rev>^!
      for-each-ref: "creator" and "creatordate" fields
      apply: handle "traditional" creation/deletion diff correctly.
      git-pickaxe: rename detection optimization
      git-pickaxe: simplify Octopus merges further
      git-pickaxe: re-scan the blob after making progress with -M
      git-pickaxe: re-scan the blob after making progress with -C
      git-pickaxe: fix origin refcounting
      cherry is built-in, do not ship git-cherry.sh
      git-blame: add internal statistics to count read blobs.
      git-pickaxe: optimize by avoiding repeated read_sha1_file().
      adjust_shared_perm: chmod() only when needed.
      Document git-pack-refs and link it to git(7).
      git-pickaxe: -L /regexp/,/regexp/
      git-pickaxe: allow "-L <something>,+N"
      GIT 1.4.3-rc1

   Linus Torvalds (2):
      Allow '-' in config variable names
      git push: add verbose flag and allow overriding of default target repository

   Nicolas Pitre (14):
      enable index-pack streaming capability
      make index-pack able to complete thin packs.
      add progress status to index-pack
      mimic unpack-objects when --stdin is used with index-pack
      enhance clone and fetch -k experience
      index-pack: minor fixes to comment and function name
      missing small substitution
      make git-push a bit more verbose
      Allow pack header preprocessing before unpack-objects/index-pack.
      git-fetch can use both --thin and --keep with fetch-pack now
      improve fetch-pack's handling of kept packs
      have index-pack create .keep file more carefully
      remove .keep pack lock files when done with refs update
      git-pack-objects progress flag documentation and cleanup

   Petr Baudis (1):
      gitweb: Support for 'forks'

   Sean Estabrooks (1):
      Add --global option to git-repo-config.

   Shawn O. Pearce (11):
      Added completion support for git-branch.exe.
      Added bash completion support for git-reset.
      Use ULONG_MAX rather than implicit cast of -1.
      Remove SIMPLE_PROGRAMS and make git-daemon a normal program.
      Remove unsupported C99 style struct initializers in git-archive.
      Added missing completions for show-branch and merge-base.
      Only load .exe suffix'd completions on Cygwin.
      Bash completion support for remotes in .git/config.
      Take --git-dir into consideration during bash completion.
      Support bash completion on symmetric difference operator.
      Remove more sed invocations from within bash completion.

   Shawn Pearce (5):
      Allow short pack names to git-pack-objects --unpacked=.
      Only repack active packs by skipping over kept packs.
      Teach git-index-pack how to keep a pack file.
      Remove unused variable in receive-pack.
      Teach receive-pack how to keep pack files based on object count.

   Tero Roponen (1):
      remove an unneeded test


* The 'next' branch, in addition, has these.

   Junio C Hamano:
      upload-pack: stop the other side when they have more roots than we do.


* The 'pu' branch, in addition, has these.

   Johannes Schindelin (6):
      Build in shortlog
      upload-pack: no longer call rev-list
      support fetching into a shallow repository
      allow cloning a repository "shallowly"
      allow deepening of a shallow repository
      add tests for shallow stuff

   Junio C Hamano (6):
      git-branch -a: show both local and remote tracking branches.
      git-commit: show --summary after successful commit.
      para-walk: walk n trees, index and working tree in parallel
      git-diff/git-apply: make diff output a bit friendlier to GNU patch (part 2)
      rev-list --left-right
      blame and pickaxe: --show-stats for easier optimization work.

Re: What's in git.git

[David Lang]

On Tue, 7 Nov 2006, Junio C Hamano wrote:

> [pu]
>
>  Johannes's shallow clone work now should rebase cleanly on top
>  of 'master' although I haven't done so yet.  As he said
>  himself the series is waiting for people who have needs for
>  such a feature to raise hands.

I haven't been watching this recently, but if this is what I understand it to be 
(the ability to get a partial repository from upstream and work normally from 
there with the result of data-mineing tools sometimes reporting 'that's part of 
the truncated history' if they hit the cutoff) consider my hand raised.

there are a number of cases where I would be interested in following a project 
as it moves forwards, but do not have the need to have the full history (even 
with the good compression that a git pack provides, it's still a significant 
amount of disk space and download time for large projects)

Re: What's in git.git

[Jakub Narebski]

Junio C Hamano wrote:

>    Jakub Narebski (4):
[...]
>       gitweb: Output also empty patches in "commitdiff" view

I think that this patch is a bit premature. Without "new improved patchset
view" the empty patches are just that: totally empty. It is new header and
especially outputting extended header information which makes outputting
"empty" patches (with no diff) in "commitdiff" view usefull. By "empty"
patches I mean pure type change, pure rename, pure copy and type change and
rename.

The "new improved patchset view" is prepared to send in two stages: updated
filename quoting/unquoting series (two patches: unescape + minimal
esc_path; esc_path with span.cntrl and backslash sequence quoting), and
"new improved patchset view" (three patches originally, two remains: using
"git diff" header and extended git diff headers; output and link also to
patches with empty diff; improved formatting of chunk header).
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: What's in git.git

[Jakub Narebski]

Junio C Hamano wrote:

> [master]
> 
>   Three topics that have been cooking in 'next' have been
>   merged, I've tagged the tip as v1.4.4-rc1.

By the way, tag v1.4.4-rc1 has "GIT 1.4.4-rc1" as title, while the commit it
points to, v1.4.4-rc1^{} has "GIT 1.4.3-rc1" as a title.
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: What's in git.git

[Junio C Hamano]

Jakub Narebski <jnareb@gmail.com> writes:

> Junio C Hamano wrote:
>
>>    Jakub Narebski (4):
> [...]
>>       gitweb: Output also empty patches in "commitdiff" view
>
> I think that this patch is a bit premature. Without "new improved patchset
> view" the empty patches are just that: totally empty. It is new header and
> especially outputting extended header information which makes outputting
> "empty" patches (with no diff) in "commitdiff" view usefull. By "empty"
> patches I mean pure type change, pure rename, pure copy and type change and
> rename.
>
> The "new improved patchset view" is prepared to send in two stages...

At least this does not break the page even without these two
follow-ups.  If the follow-ups come in time and do not break the
page, they are very welcome post -rc1 fixes to have before the
final release.  On the other hand, even if they don't, it's not
the end of the world ;-).

Re: What's in git.git

[Junio C Hamano]

Jakub Narebski <jnareb@gmail.com> writes:

> Junio C Hamano wrote:
>
>> [master]
>> 
>>   Three topics that have been cooking in 'next' have been
>>   merged, I've tagged the tip as v1.4.4-rc1.
>
> By the way, tag v1.4.4-rc1 has "GIT 1.4.4-rc1" as title, while the commit it
> points to, v1.4.4-rc1^{} has "GIT 1.4.3-rc1" as a title.

Yeah, sometimes I make typoes.  Not a news X-<.

Re: What's in git.git

[Petr Baudis]

>   By the way, do people mind if I start to rewind and rebase
>   'next' after every feature release (i.e. tagged release is
>   made after 'master')?  I do not feel a strong need for it, and
>   'git log --no-merges master..next' will show emptiness
>   eventually, but being able to restart from clean slate after a
>   release would be somewhat nice.

  It would be annoying for me since I can't mantain a long-term private
branch tracking next then (and I prefer next to master). I don't do that
right now but I've done that shortly in the past and it would be nice to
have that possibility in the future as well.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
#!/bin/perl -sp0777i<X+d*lMLa^*lN%0]dsXx++lMlN/dsM0<j]dsj
$/=unpack('H*',$_);$_=`echo 16dio\U$k"SK$/SM$n\EsN0p[lN*1

Shallow clone [Was Re: What's in git.git ]

[Aneesh Kumar K.V]

David Lang wrote:
> On Tue, 7 Nov 2006, Junio C Hamano wrote:
> 
>> [pu]
>>
>>  Johannes's shallow clone work now should rebase cleanly on top
>>  of 'master' although I haven't done so yet.  As he said
>>  himself the series is waiting for people who have needs for
>>  such a feature to raise hands.
> 
> I haven't been watching this recently, but if this is what I understand 
> it to be (the ability to get a partial repository from upstream and work 
> normally from there with the result of data-mineing tools sometimes 
> reporting 'that's part of the truncated history' if they hit the cutoff) 
> consider my hand raised.
> 
> there are a number of cases where I would be interested in following a 
> project as it moves forwards, but do not have the need to have the full 
> history (even with the good compression that a git pack provides, it's 
> still a significant amount of disk space and download time for large 
> projects)
> 

I am trying to test this feature. Is there a documentation .git/shallow some where. Atleast what those entries
mean ? I know in the mail johannes mentioned only core git will touch this file. But it should be ok to be 
descriptive like other files. (FETCH_HEAD)

-aneesh 

Re: Shallow clone [Was Re: What's in git.git ]

[Aneesh Kumar K.V]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

Aneesh Kumar K.V wrote:
> David Lang wrote:
>> On Tue, 7 Nov 2006, Junio C Hamano wrote:
>>
>>> [pu]
>>>
>>>  Johannes's shallow clone work now should rebase cleanly on top
>>>  of 'master' although I haven't done so yet.  As he said
>>>  himself the series is waiting for people who have needs for
>>>  such a feature to raise hands.
>>
>> I haven't been watching this recently, but if this is what I 
>> understand it to be (the ability to get a partial repository from 
>> upstream and work normally from there with the result of data-mineing 
>> tools sometimes reporting 'that's part of the truncated history' if 
>> they hit the cutoff) consider my hand raised.
>>
>> there are a number of cases where I would be interested in following a 
>> project as it moves forwards, but do not have the need to have the 
>> full history (even with the good compression that a git pack provides, 
>> it's still a significant amount of disk space and download time for 
>> large projects)
>>
> 
> I am trying to test this feature. Is there a documentation .git/shallow 
> some where. Atleast what those entries
> mean ? I know in the mail johannes mentioned only core git will touch 
> this file. But it should be ok to be descriptive like other files. 
> (FETCH_HEAD)


How about this 

-aneesh 

[-- Attachment #2: repository-layout.txt.diff --]
[-- Type: text/x-patch, Size: 537 bytes --]

diff --git a/Documentation/repository-layout.txt b/Documentation/repository-layout.txt
index 275d18b..03a6f77 100644
--- a/Documentation/repository-layout.txt
+++ b/Documentation/repository-layout.txt
@@ -141,3 +141,9 @@ logs/refs/heads/`name`::
 
 logs/refs/tags/`name`::
 	Records all changes made to the tag named `name`.
+
+shallow::
+	Records the sha1 of the commits which is marked to have no
+	parents to represent a shallow repository.The commit object
+	will have the parent information present. It carry one
+	record per line.

Re: What's in git.git

[Junio C Hamano]

Sorry, I made a mistake of pushing out git-pickaxe before making
it take over git-blame.  So I will fix it up by renaming it to
git-blame.

That is, unless there are too many people whose fingers have
already been trained to type "git-pickaxe" from 'next'
experience (you can obviously count me as one of these people).
In which case I will keep git-pickaxe as a backward compatible
synonym just like we still have git-annotate.

It might also make sense to eventually remove the other synonym
git-annotate but that would be a longer term issue and should be
handled separately.

Re: What's in git.git

[Horst H. von Brand]

David Lang <dlang@digitalinsight.com> wrote:
> On Tue, 7 Nov 2006, Junio C Hamano wrote:
> 
> > [pu]
> >
> >  Johannes's shallow clone work now should rebase cleanly on top
> >  of 'master' although I haven't done so yet.  As he said
> >  himself the series is waiting for people who have needs for
> >  such a feature to raise hands.
> 
> I haven't been watching this recently, but if this is what I
> understand it to be (the ability to get a partial repository from
> upstream and work normally from there with the result of data-mineing
> tools sometimes reporting 'that's part of the truncated history' if
> they hit the cutoff) consider my hand raised.

+1
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                    Fono: +56 32 2654431
Universidad Tecnica Federico Santa Maria             +56 32 2654239
Casilla 110-V, Valparaiso, Chile               Fax:  +56 32 2797513

Re: What's in git.git

[Junio C Hamano]

"Horst H. von Brand" <vonbrand@laptop13.inf.utfsm.cl> writes:

> David Lang <dlang@digitalinsight.com> wrote:
>> On Tue, 7 Nov 2006, Junio C Hamano wrote:
>> 
>> > [pu]
>> >
>> >  Johannes's shallow clone work now should rebase cleanly on top
>> >  of 'master' although I haven't done so yet.  As he said
>> >  himself the series is waiting for people who have needs for
>> >  such a feature to raise hands.
>> 
>> I haven't been watching this recently, but if this is what I
>> understand it to be (the ability to get a partial repository from
>> upstream and work normally from there with the result of data-mineing
>> tools sometimes reporting 'that's part of the truncated history' if
>> they hit the cutoff) consider my hand raised.
>
> +1

What does that plus one mean?  I do not know where people picked
up this annoying plus or minus one business, but could you all
stop that?

If you are volunteering to help debugging and feeding bugfixes
that is very much welcome and appreciated.

Thanks.

Re: What's in git.git

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> "Horst H. von Brand" <vonbrand@laptop13.inf.utfsm.cl> writes:
>
>>> On Tue, 7 Nov 2006, Junio C Hamano wrote:
>>> 
>>> > [pu]
>>> >
>>> >  Johannes's shallow clone work now should rebase cleanly on top
>>> >  of 'master' although I haven't done so yet.  As he said
>>> >  himself the series is waiting for people who have needs for
>>> >  such a feature to raise hands.
>>
>> +1
>
> What does that plus one mean?  I do not know where people picked
> up this annoying plus or minus one business, but could you all
> stop that?
>
> If you are volunteering to help debugging and feeding bugfixes
> that is very much welcome and appreciated.

Sorry, I sent the message without finishing.  What should have
followed is...

  On the other hand, if you are saying "yes, shallow would fit
  my workflow and would be useful to have, but I do not have
  time nor inclination to help hacking on it", that is also
  fine.

  But "+1" does not help me tell which one you really mean.

> Thanks.

Re: What's in git.git

[Dave Dillow]

On Wed, 2006-11-08 at 18:54 -0800, Junio C Hamano wrote:
> "Horst H. von Brand" <vonbrand@laptop13.inf.utfsm.cl> writes:
> 
> > David Lang <dlang@digitalinsight.com> wrote:
> >> On Tue, 7 Nov 2006, Junio C Hamano wrote:
> >> 
> >> > [pu]
> >> >
> >> >  Johannes's shallow clone work now should rebase cleanly on top
> >> >  of 'master' although I haven't done so yet.  As he said
> >> >  himself the series is waiting for people who have needs for
> >> >  such a feature to raise hands.
> >> 
> >> I haven't been watching this recently, but if this is what I
> >> understand it to be (the ability to get a partial repository from
> >> upstream and work normally from there with the result of data-mineing
> >> tools sometimes reporting 'that's part of the truncated history' if
> >> they hit the cutoff) consider my hand raised.
> >
> > +1
> 
> What does that plus one mean?  I do not know where people picked
> up this annoying plus or minus one business, but could you all
> stop that?

Horst can speak for himself, but I'd wager he's using the Apache voting
conventions:

Re: Shallow clone

[Junio C Hamano]

"Aneesh Kumar K.V" <aneesh.kumar@gmail.com> writes:

>> I am trying to test this feature. Is there a documentation
>> .git/shallow some where. Atleast what those entries
>> mean ? I know in the mail johannes mentioned only core git will
>> touch this file. But it should be ok to be descriptive like other
>> files. (FETCH_HEAD)
>
> diff --git a/Documentation/repository-layout.txt b/Documentation/repository-layout.txt
> index 275d18b..03a6f77 100644
> --- a/Documentation/repository-layout.txt
> +++ b/Documentation/repository-layout.txt
> @@ -141,3 +141,9 @@ logs/refs/heads/`name`::
>  
>  logs/refs/tags/`name`::
>  	Records all changes made to the tag named `name`.
> +
> +shallow::
> +	Records the sha1 of the commits which is marked to have no
> +	parents to represent a shallow repository.The commit object
> +	will have the parent information present. It carry one
> +	record per line.

I would drop the second sentence which is just confusing but
otherwise it is correct, I think (I just started trying it
out).

But it seems to need some more work.  I just tried to clone
git.git with --depth=1 and it cauterizes each branch with two
commits (I think that is what depth=1 means -- the latest and
one behind it), but it pulled almost the whole repository
anyway, and it turns out that "git log v1.4.3-rc1" gives me the
full history leading to it.

Subsequent "git fetch --depth 99999" makes the branches
connected to the root commit, and I am reasonably sure we do not
have that many commits, but .git/shallow did not become empty.

I haven't followed the code closely enough to tell if these are
just minor details needing more polish, or something more
fundamental in the design.

Re: Shallow clone

[Aneesh Kumar]

On 11/9/06, Junio C Hamano <junkio@cox.net> wrote:
> "Aneesh Kumar K.V" <aneesh.kumar@gmail.com> writes:
>
> >
> But it seems to need some more work.  I just tried to clone
> git.git with --depth=1 and it cauterizes each branch with two
> commits (I think that is what depth=1 means -- the latest and
> one behind it), but it pulled almost the whole repository
> anyway, and it turns out that "git log v1.4.3-rc1" gives me the
> full history leading to it.
>
> Subsequent "git fetch --depth 99999" makes the branches
> connected to the root commit, and I am reasonably sure we do not
> have that many commits, but .git/shallow did not become empty.
>

I tried a local clone with --depth 2 and it showed two commits
in all the branches/DAG. But then doing a  pull --depth 4 listed more
commits than 4. I didn't look much into this.

Re: Shallow clone

[Alexandre Julliard]

Junio C Hamano <junkio@cox.net> writes:

> But it seems to need some more work.  I just tried to clone
> git.git with --depth=1 and it cauterizes each branch with two
> commits (I think that is what depth=1 means -- the latest and
> one behind it), but it pulled almost the whole repository
> anyway, and it turns out that "git log v1.4.3-rc1" gives me the
> full history leading to it.

That's apparently because tags are not considered when truncating the
commit list. The patch below fixes it, and fetches the right number of
commits for each tag. However the correct fix is probably to not fetch
historical tags at all.

There's also a problem with the packing, a clone --depth 1 currently
results in a pack that's about 3 times as large as it should be.

---
diff --git a/shallow.c b/shallow.c
index 58a7b20..2db1dc4 100644
--- a/shallow.c
+++ b/shallow.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "commit.h"
+#include "tag.h"
 
 static int is_shallow = -1;
 
@@ -54,7 +55,7 @@ struct commit_list *get_shallow_commits(
 		if (!commit) {
 			if (i < heads->nr) {
 				commit = (struct commit *)
-					heads->objects[i++].item;
+					deref_tag(heads->objects[i++].item, NULL, 0);
 				if (commit->object.type != OBJ_COMMIT) {
 					commit = NULL;
 					continue;

-- 
Alexandre Julliard

Re: Shallow clone

[Junio C Hamano]

Alexandre Julliard <julliard@winehq.org> writes:

> There's also a problem with the packing, a clone --depth 1 currently
> results in a pack that's about 3 times as large as it should be.

That's interesting.

  : gitster; git clone -n --depth 1 git://127.0.0.1/git.git victim-001
  remote: Generating pack...
  remote: Done counting 6246 objects.
  remote: Deltifying 6246 objects.
  remote:  100% (6246/6246) done
  Indexing 6246 objects.
  remote: Total 6246, written 6246 (delta 3106), reused 4313 (delta 3106)
   100% (6246/6246) done
  Resolving 3106 deltas.
   100% (3106/3106) done
  : gitster; cd victim-001
  : gitster; ls -lh .git/objects/pack/
  total 9.6M
  drwxrwsr-x 2 junio src 4.0K 2006-11-11 23:52 ./
  drwxrwsr-x 4 junio src 4.0K 2006-11-11 23:52 ../
  -r--r--r-- 1 junio src 148K 2006-11-11 23:52 pack-f5f88d83....idx
  -r--r--r-- 1 junio src 9.5M 2006-11-11 23:52 pack-f5f88d83....pack

Repacking immediately after cloning brings it down to what is
expected.

  : gitster; git repack -a -d -f
  Generating pack...
  Done counting 6246 objects.
  Deltifying 6246 objects.
   100% (6246/6246) done
  Writing 6246 objects.
   100% (6246/6246) done
  Total 6246, written 6246 (delta 4815), reused 1407 (delta 0)
  Pack pack-f5f88d83524213e3ab05697ff75f245b1ef9081a created.
  : gitster; ls -lh .git/objects/pack/
  total 2.8M
  drwxrwsr-x 2 junio src 4.0K 2006-11-11 23:53 ./
  drwxrwsr-x 4 junio src 4.0K 2006-11-11 23:52 ../
  -rw-rw-r-- 1 junio src 148K 2006-11-11 23:53 pack-f5f88d83....idx
  -rw-rw-r-- 1 junio src 2.6M 2006-11-11 23:53 pack-f5f88d83....pack

In any case, after this "shallow" stuff, repeated "fetch --depth
99" seems to fetch 0 object and 3400 objects alternately, and
the shallow file alternates between 900 bytes and 11000 bytes.

We would need to take a deeper look into what this series does,
before moving it to 'next'.

Re: Shallow clone [Was Re: What's in git.git ]

[Petr Baudis]

On Wed, Nov 08, 2006 at 06:59:05PM CET, Aneesh Kumar K.V wrote:
> diff --git a/Documentation/repository-layout.txt b/Documentation/repository-layout.txt
> index 275d18b..03a6f77 100644
> --- a/Documentation/repository-layout.txt
> +++ b/Documentation/repository-layout.txt
> @@ -141,3 +141,9 @@ logs/refs/heads/`name`::
>  
>  logs/refs/tags/`name`::
>  	Records all changes made to the tag named `name`.
> +
> +shallow::

Hmm, shouldn't the shallow file rather belong to the info/ subdirectory?

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
#!/bin/perl -sp0777i<X+d*lMLa^*lN%0]dsXx++lMlN/dsM0<j]dsj
$/=unpack('H*',$_);$_=`echo 16dio\U$k"SK$/SM$n\EsN0p[lN*1

Re: Shallow clone

[Sergey Vlasov]

[-- Attachment #1: Type: text/plain, Size: 2619 bytes --]

On Sun, 12 Nov 2006 00:16:40 -0800 Junio C Hamano wrote:

> Alexandre Julliard <julliard@winehq.org> writes:
>
> > There's also a problem with the packing, a clone --depth 1 currently
> > results in a pack that's about 3 times as large as it should be.
>
> That's interesting.
>
>   : gitster; git clone -n --depth 1 git://127.0.0.1/git.git victim-001
[...]
>   -r--r--r-- 1 junio src 9.5M 2006-11-11 23:52 pack-f5f88d83....pack
>
> Repacking immediately after cloning brings it down to what is
> expected.
>
>   : gitster; git repack -a -d -f
[...]
>   -rw-rw-r-- 1 junio src 2.6M 2006-11-11 23:53 pack-f5f88d83....pack

This is due to optimization in builtin-pack-objects.c:try_delta():

	/*
	 * We do not bother to try a delta that we discarded
	 * on an earlier try, but only when reusing delta data.
	 */
	if (!no_reuse_delta && trg_entry->in_pack &&
	    trg_entry->in_pack == src_entry->in_pack)
		return 0;

After removing this part the shallow pack after clone is 2.6M, as it
should be.

The problem with this optimization is that it is only valid if we are
repacking either the same set of objects as we did earlier, or its
superset.  But if we are packing a subset of objects, there will be some
objects in that subset which were delta-compressed in the original pack,
but base objects for that deltas are not included in our subset -
therefore we will be unable to reuse existing deltas, and with that
optimization we will never try to use delta compression for these
objects.  (The optimization assumes that if we will try to use delta
compression, we will try mostly the same base objects as we have tried
when we made the existing pack, and therefore will likely get the same
result - which is close to the truth when we are doing "repack -a", but
is badly wrong when we are doing "git-upload-pack" with a large number
of common commits, and therefore are excluding a lot of objects.)

So any partial fetch (shallow or not) from a mostly packed repository
currently results in a suboptimal pack.  In fact, the fresh "repack -a
-d -f" is probably the worst case for subsequent fetch (not initial
clone) from that repository - objects for the most recent commit are
most likely to be stored without delta compression, and even if deltas
are used, they are likely in the wrong direction for someone who has an
older version and wants to update it.


> In any case, after this "shallow" stuff, repeated "fetch --depth
> 99" seems to fetch 0 object and 3400 objects alternately, and
> the shallow file alternates between 900 bytes and 11000 bytes.

I confirm this - different numbers, but the same problem...

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Shallow clone

[Junio C Hamano]

Petr Baudis <pasky@suse.cz> writes:

> On Wed, Nov 08, 2006 at 06:59:05PM CET, Aneesh Kumar K.V wrote:
>> diff --git a/Documentation/repository-layout.txt b/Documentation/repository-layout.txt
>> index 275d18b..03a6f77 100644
>> --- a/Documentation/repository-layout.txt
>> +++ b/Documentation/repository-layout.txt
>> @@ -141,3 +141,9 @@ logs/refs/heads/`name`::
>>  
>>  logs/refs/tags/`name`::
>>  	Records all changes made to the tag named `name`.
>> +
>> +shallow::
>
> Hmm, shouldn't the shallow file rather belong to the info/ subdirectory?

I think that makes sense.

Re: Shallow clone

[Junio C Hamano]

Sergey Vlasov <vsu@altlinux.ru> writes:

> This is due to optimization in builtin-pack-objects.c:try_delta():
>
> 	/*
> 	 * We do not bother to try a delta that we discarded
> 	 * on an earlier try, but only when reusing delta data.
> 	 */
> 	if (!no_reuse_delta && trg_entry->in_pack &&
> 	    trg_entry->in_pack == src_entry->in_pack)
> 		return 0;
>
> After removing this part the shallow pack after clone is 2.6M, as it
> should be.
>
> The problem with this optimization is that it is only valid if we are
> repacking either the same set of objects as we did earlier, or its
> superset.  But if we are packing a subset of objects, there will be some
> objects in that subset which were delta-compressed in the original pack,
> but base objects for that deltas are not included in our subset -
> therefore we will be unable to reuse existing deltas, and with that
> optimization we will never try to use delta compression for these
> objects.
> ...
> So any partial fetch (shallow or not) from a mostly packed repository
> currently results in a suboptimal pack.

That is correct.  How about something like this?

I think the determination of "repacking_superset" may need to be
tweaked because existing packs may have overlaps, and the patch
counts them once per pack.


diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index 69e5dd3..fb25124 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -64,6 +64,7 @@ struct object_entry {
 static unsigned char object_list_sha1[20];
 static int non_empty;
 static int no_reuse_delta;
+static int repacking_superset;
 static int local;
 static int incremental;
 static int allow_ofs_delta;
@@ -1172,10 +1173,13 @@ static int try_delta(struct unpacked *tr
 		return -1;
 
 	/*
-	 * We do not bother to try a delta that we discarded
-	 * on an earlier try, but only when reusing delta data.
+	 * When we are packing the superset of objects we have already
+	 * packed, we do not bother to try a delta that we discarded
+	 * on an earlier try.  This heuristic of course should not
+	 * kick in when we are not reusing delta, or we know we are
+	 * sending a subset of objects from a repository.
 	 */
-	if (!no_reuse_delta && trg_entry->in_pack &&
+	if (!no_reuse_delta && repacking_superset && trg_entry->in_pack &&
 	    trg_entry->in_pack == src_entry->in_pack)
 		return 0;
 
@@ -1493,6 +1497,16 @@ static void get_object_list(int ac, cons
 	traverse_commit_list(&revs, show_commit, show_object);
 }
 
+static int count_packed_objects(void)
+{
+	struct packed_git *p;
+	int cnt = 0;
+
+	for (p = packed_git; p; p = p->next)
+		cnt += num_packed_objects(p);
+	return cnt;
+}
+
 int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 {
 	SHA_CTX ctx;
@@ -1631,6 +1645,8 @@ int cmd_pack_objects(int argc, const cha
 	if (non_empty && !nr_result)
 		return 0;
 
+	repacking_superset = count_packed_objects() < nr_result;
+
 	SHA1_Init(&ctx);
 	list = sorted_by_sha;
 	for (i = 0; i < nr_result; i++) {

Re: What's in git.git

[Johannes Schindelin]

Hi,

On Tue, 7 Nov 2006, David Lang wrote:

> On Tue, 7 Nov 2006, Junio C Hamano wrote:
> 
> > [pu]
> > 
> >  Johannes's shallow clone work now should rebase cleanly on top
> >  of 'master' although I haven't done so yet.  As he said
> >  himself the series is waiting for people who have needs for
> >  such a feature to raise hands.
> 
> I haven't been watching this recently, but if this is what I understand it to
> be (the ability to get a partial repository from upstream and work normally
> from there with the result of data-mineing tools sometimes reporting 'that's
> part of the truncated history' if they hit the cutoff) consider my hand
> raised.

For now, it does not say "part of the truncated history". But yes, shallow 
clones are partial copies of remote repositories, by making some commits 
"shallow", i.e. grafting an empty set of parents onto them (thereby 
pretending that these commits are root commits).

Telling the user that a commit is shallow should not be too hard.

Ciao,
Dscho

Re: Shallow clone

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> Sergey Vlasov <vsu@altlinux.ru> writes:
>
>> This is due to optimization in builtin-pack-objects.c:try_delta():
>>
>> 	/*
>> 	 * We do not bother to try a delta that we discarded
>> 	 * on an earlier try, but only when reusing delta data.
>> 	 */
>> 	if (!no_reuse_delta && trg_entry->in_pack &&
>> 	    trg_entry->in_pack == src_entry->in_pack)
>> 		return 0;
>>
>> After removing this part the shallow pack after clone is 2.6M, as it
>> should be.
>>
>> The problem with this optimization is that it is only valid if we are
>> repacking either the same set of objects as we did earlier, or its
>> superset.  But if we are packing a subset of objects, there will be some
>> objects in that subset which were delta-compressed in the original pack,
>> but base objects for that deltas are not included in our subset -
>> therefore we will be unable to reuse existing deltas, and with that
>> optimization we will never try to use delta compression for these
>> objects.
>> ...
>> So any partial fetch (shallow or not) from a mostly packed repository
>> currently results in a suboptimal pack.

What we tried to avoid with the original heuristics in commit
51d1e83f was to avoid wasting time on undeltifiable blobs, and
they would be stored as base in the original packs, so I think
the following would fly better (the patch is for maint, for
master we would also check for OBJ_REF_DELTA as well).

---

diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index 96c069a..84a8749 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -1101,7 +1101,8 @@ static int try_delta(struct unpacked *tr
 	 * on an earlier try, but only when reusing delta data.
 	 */
 	if (!no_reuse_delta && trg_entry->in_pack &&
-	    trg_entry->in_pack == src_entry->in_pack)
+	    trg_entry->in_pack == src_entry->in_pack &&
+	    trg_entry->in_pack_type != OBJ_DELTA)
 		return 0;
 
 	/*