GIT 0.99.9c

GIT 0.99.9c

[Junio C Hamano]

GIT 0.99.9c is found at http://kernel.org/pub/software/scm/git/
as usual.  It includes the following fixes and documentation
updates since 0.99.9b:

	Alex Riesen:
	      remove CR/LF from .gitignore

	Jon Loeliger
	      Illustration: "Fundamental Git Index Operations"
	      Illustration: "Git Diff Types"
	      Illustration: "Commit DAG Revision Naming"

	Junio C Hamano:
	      Do not put automatic merge message after signed-off-by line.
	      git-clone: do not forget to create origin branch.
	      Make test-date buildable again.
	      Do not fail on hierarchical branch names.
	      Ignore '\r' at the end of line in $GIT_DIR/config
	      Be careful when dereferencing tags (credits Pasky).
	      Document --since and --until options to rev-parse.
	      Add --no-commit to git-merge/git-pull.
	      Add 'ours' merge strategy.
	      git-merge-ours: make sure our index matches HEAD
	      GIT 0.99.9c

	Peter Eriksen:
	      Clean up the SunOS Makefile rule

The slow and steady march toward 1.0 continues.  

I plan to do another full sweep in the documentation directory
on my next GIT day.

On the proposed updates front, I am hoping to include Nick's
http-push using DAV in the "master" branch soon.  And I would
appreciate somebody who actually uses svnimport to Ack on
Yaacov's svnimport fix.

RFC: GIT networked storage

[Jeff Garzik]

Here's an experiment I've been dying to try.

The current "tracker-less" BitTorrent[1] employs a distributed hash 
table[2] called Kademlia, where the total content is spread across a 
bunch of computers on the network.  I kinda prefer TANGLE[3] to Kademlia.

Anyway, I was thinking that it would be a neat experiment to add simple 
TANGLE-like peer-to-peer code, to enable git to query "the git network 
hash table" for content.

Comments, or any pre-code-creation objections?
How easy is it to add a new storage backend to git?

To restrict unlimited uploading, I'm thinking that I'll want the system 
to fall back to {www,git,rsync}.kernel.org as the original source of 
content.  [though the code will obviously be generic, and not hardcode 
*.kernel.org policy]

Thanks,

	Jeff



[1] http://www.bittorrent.com/trackerless.html
[2] http://www.etse.urv.es/~cpairot/dhts.html
[3] http://www.nicemice.net/amc/research/tangle/

Re: RFC: GIT networked storage

[Junio C Hamano]

Jeff Garzik <jgarzik@pobox.com> writes:

> How easy is it to add a new storage backend to git?

Almost everything is contained within sha1_file.c.

Object creation side is simple -- everybody who creates an
object (e.g update-index registering blobs, write-tree writing
the toplevel and intermediate level trees, commit-tree building
a commit object, unpack-objects exploding a pack) goes through
write_sha1_file(), which checks if the object is already
available using has_sha1_file() and creates a new object in the
local .git/objects/?? directory.  I am assuming that you are not
planning to create objects in a remote peer from within the git
code path, and instead to have background process that replicate
them over the network to peer repositories, so you probably do
not have to touch this side.

Extending inspection and reading from existing objects for your
networked storage may be somewhat messy, but starting points
are:

. has_sha1_file() takes the object name and returns true/false;
  if the object is available to us in any form (be it in one of
  the alternate object stores or the local repository, as
  individual object in an objects/??/ file or stored in a pack).
  Currently it looks at packs and then checks individual files
  for performance reasons (to minimize seeks and to prevent
  polluting dcache with many negative hits); you would be adding
  another data source. 

. read_sha1_file() takes the object name, and returns the
  uncompressed contents of the object in addition to the type
  and the size.

. sha1_object_info() takes the object name and returns the type
  of the object and optionally returns the size of it, without
  reading the contents.  Some programs call this before reading
  the data using read_sha1_file(), so you might want to use this
  as a cue to prefetch from a remote peer; also you _might_ want
  to keep type and size cached if you plan to implement
  forgetful storage that deliberately loses objects and expects
  to refetch it from its peers.  Good test program once you are
  done extending this part is git-cat-file with -t and -s flag.

The above three are the primary read interfaces, but there are
some places that cheat by assuming that packs and individual
objects are the only two kinds of sources for the object data,
so you need to be careful.  For example, write_sha1_to_fd(),
which is used only by ssh-upload, first tries to call
map_sha1_file_internal(), which is only valid for individual
objects, to grab the object data, and when it fails, calls
read_packed_sha1(), which is only valid for objects in packs,
without even checking if read_packed_sha1() succeeded.  This
doesn't crash only because the caller of write_sha1_to_fd()
checks if the object is available by calling has_sha1_file()
itself before calling this function, but you would need to
change it to fall back on your networked storage if you do not
want to crash when used by ssh-upload.

HTH, and have fun.

Re: RFC: GIT networked storage

[Jeff Garzik]

Junio C Hamano wrote:
> Jeff Garzik <jgarzik@pobox.com> writes:
> 
> 
>>How easy is it to add a new storage backend to git?
> 
> 
> Almost everything is contained within sha1_file.c.
> 
> Object creation side is simple -- everybody who creates an
> object (e.g update-index registering blobs, write-tree writing
> the toplevel and intermediate level trees, commit-tree building
> a commit object, unpack-objects exploding a pack) goes through
> write_sha1_file(), which checks if the object is already
> available using has_sha1_file() and creates a new object in the
> local .git/objects/?? directory.  I am assuming that you are not
> planning to create objects in a remote peer from within the git
> code path, and instead to have background process that replicate
> them over the network to peer repositories, so you probably do
> not have to touch this side.
> 
> Extending inspection and reading from existing objects for your
> networked storage may be somewhat messy, but starting points
> are:
[...]

Thanks.  It looks pretty straightforward to add a new storage backend, 
grepping on the symbols you listed.

	Jeff

Re: GIT 0.99.9c

[Santi Béjar]

Junio C Hamano <junkio@cox.net> writes:

>  And I would appreciate somebody who actually uses svnimport to Ack on
> Yaacov's svnimport fix.

Let's try to make a good report :)

Situation:

an svn repo created with cvs2svn, and used with svk so it has
the property svk:merge to say explicity the extra parents of a commit.

svn log -r 1:2 -v file:///repo/susy/
------------------------------------------------------------------------
r1 | (no author) | 2003-10-29 16:49:04 +0100 (Wed, 29 Oct 2003) | 1 line
Changed paths:
   A /branches
   A /tags
   A /trunk

New repository initialized by cvs2svn.
------------------------------------------------------------------------
r2 | guasch | 2003-10-29 16:49:04 +0100 (Wed, 29 Oct 2003) | 2 lines
Changed paths:
   A /trunk/includes
   A /trunk/includes/gamma_gg.F

Initial revision

------------------------------------------------------------------------

git-svnimport orig:

bela $ git-svnimport -v file:///repo/susy/
1: Unrecognized path: /branches
1: Unrecognized path: /tags
Tree ID 4b825dc642cb6eb9a060e54bf8d69288fbee4904
Committing initial tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904
Committed change 1:/ 2003-10-29 15:49:04)
Commit ID caff0f3ee5bc540a264f615b000fcbd7b355586a
Writing to refs/heads/origin
DONE: 1 origin caff0f3ee5bc540a264f615b000fcbd7b355586a
... 2 /trunk/includes ...
Name does not refer to a filesystem file: Attempted to get textual contents of a *non*-file node at /home/santi/usr/bin/git-svnimport line 115
bela $ git-cat-file tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904
bela $ 

and the same error for the commit r2 with "-s 2". I suppose that it
tries to chechkout the includes dir.

git-svnimport new:

I've used the one in the message
"[PATCH] Several fixes to import mono's svn tree"

bela $ git-svnimport -v file:///repo/susy/
Tree ID 4b825dc642cb6eb9a060e54bf8d69288fbee4904
Committing initial tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904
Committed change 1:/ 2003-10-29 15:49:04)
Commit ID caff0f3ee5bc540a264f615b000fcbd7b355586a
Writing to refs/heads/origin
DONE: 1 origin caff0f3ee5bc540a264f615b000fcbd7b355586a
... 2 /trunk/includes/gamma_gg.F ...
Tree ID b7ea8a91831deea3a6015a17f8843be4833255bc
Merge parent branch: caff0f3ee5bc540a264f615b000fcbd7b355586a
Committed change 2:/ 2003-10-29 15:49:04)
Commit ID 6dc8146789badce6368705f9317e7f65f4526f6b
Writing to refs/heads/origin
DONE: 2 origin 6dc8146789badce6368705f9317e7f65f4526f6b

[...]

DONE: 154 SinCosAlpha 657abeb912a00d6fad9fcd4436490d81b4002bcb
Switching from 657abeb912a00d6fad9fcd4436490d81b4002bcb to 760501ff6986134eaf0ee34484af1e38e7f8b8b0 (/)
perl: /home/devel/release/subversion-1.2.3dfsg1/subversion/libsvn_subr/path.c:115: svn_path_join: Assertion `is_canonical (component, clen)' failed.
Aborted

So it worked till the revision 154. But the revision 155 is:

bela $ svn log -r 155 file:///repo/susy/
------------------------------------------------------------------------
r155 | guasch | 2005-02-25 11:19:38 +0100 (Fri, 25 Feb 2005) | 14 lines
Changed paths:
   M /
   M /trunk
   M /trunk/includes/pp.f
   A /trunk/includes/subh.F

 r157@jgi:  guasch | 2005-02-25 11:19:23 +0100
  r156@jgi:  guasch | 2005-02-25 11:17:06 +0100
  Create local branch of SVN mirror

*** Merge branch SinCosAlpha. Revisions 152:154

bela $ svn diff -r 154:155 file:///repo/susy/

[...]

Property changes on: trunk
___________________________________________________________________
Name: svk:merge
   + e2421932-edf0-0310-b7cb-9d7cda74567f:/local/branches/SinCosAlpha:156


Property changes on: .
___________________________________________________________________
Name: svk:merge
   + e2421932-edf0-0310-b7cb-9d7cda74567f:/local:157

bela $ 

This is the first revision that changes the properties of / and
/trunk, and it does not handle this situation.

A usefull thing would be if it could use this information to create a
merge commit. So it could check if this information is from the
repository we are importing, and fallback to the heuristic one if it
fails. (In the case above the "e242..." UUID is from another svn
repository).

If I do it by hand, the I get the following:
156: Unrecognized path: /Xsecfile
157: Unrecognized path: /Xsecfile/bo_Hqqp/hqqpinterface_sqcd_maximize_SSB_bsg_runningmass_prod.F
157: Unrecognized path: /Xsecfile/includes/pphtt.f
157: Unrecognized path: /Xsecfile/includes/dsig_htt.F
158: Unrecognized path: /Xsecfile/includes/dsig_htt.F
perl: /home/devel/release/subversion-1.2.3dfsg1/subversion/libsvn_subr/path.c:115: svn_path_join: Assertion `is_canonical (component, clen)' failed.
Aborted

The r159 changes the properties of /trunk too. The other thing is that
finds an "Unrecognized path" (that is logical), but I have not find any
way to tell "OK, it's my fault, but treat this as branch Xsecfile".

Hope it's usefull.


     Santi