git peer-to-peer project: info needed

git peer-to-peer project: info needed

[Luke Kenneth Casson Leighton]

hi folks,

[please could you kindly cc on responses because i am subscribed with
"no mail" set]

i need some guidance on what i should be doing, to add peer-to-peer
networking to "git fetch".  i can take care of the peer-to-peer
networking side: the bit i'm unsure about is what sequence of events
are required to happen.  i'm presently looking at the use of
walker_fetch in builtin-http-fetch.c which gives me some idea.
however as i aim to implement this first in python not c, i need to be
using git via command-line only.

perhaps... does anyone know of an implementation of "git fetch" in
say... shell-script (or perhaps in python) or just absolutely any
language _other_ than c?

i believe... i believe i may need the git rev-list and git
pack-objects commands, would that be right?  rev-list gives the list
of commit revisions; pack-objects, if i can merely transfer the .pack
file and .index file over the peer-to-peer network, the job's half
done, am i right?

(the only missing step is to have a way to find the list of objects at
the remote end - i know how to do that)

following that, i can do "git unpack-objects" on the .pack file(s)

and... i've just done a test on that (manually, by running
git-pack-objects --thin --all --stdout and then git-unpack-objects)
... um.... :)  i'm missing the "tag updating", aren't i?  the objects
are now in the .git repo but they're "hanging about", as can be seen
with git fsck:

$  git fsck
dangling commit af87b49b9fbcae28ae19b86ca04af5bd4a9f6778

so... my next step iis... ermm... update the head ref?  err? :)  i see
that gitpython is simply writing [the above ref] into
.git/ref/heads/master or whereever is specified (must begin with refs)
which is kinda cheating but perfectly reasonable... err.. but wait...
i'm into what "git-pull.sh" is doing at this point, aren't i?

soo... i kinda don't have to worry about that bit, am i right?  as
long as i get to complete the "git-unpack-objects" stage, the rest can
be handled by git-pull.sh, am i right?

much obliged some answers so that i can get this done, as free
software, and provide people with a peer-to-peer distributed version
of git.

l.

Re: git peer-to-peer project: info needed

[Jonathan Nieder]

Hi Luke,

Luke Kenneth Casson Leighton wrote:

> i need some guidance on what i should be doing, to add peer-to-peer
> networking to "git fetch".  i can take care of the peer-to-peer
> networking side: the bit i'm unsure about is what sequence of events
> are required to happen.

Not sure if this is the best answer or not, but you might enjoy
looking at the git-remote-helpers(7) feature and git-remote-testgit.py
example.

Also, in case it is useful: the old scripted "git fetch" is available
as contrib/examples/git-fetch.sh.

Hope that helps,
Jonathan

Re: git peer-to-peer project: info needed

[Luke Kenneth Casson Leighton]

On Sun, Aug 29, 2010 at 11:06 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Hi Luke,
>
> Luke Kenneth Casson Leighton wrote:
>
>> i need some guidance on what i should be doing, to add peer-to-peer
>> networking to "git fetch".  i can take care of the peer-to-peer
>> networking side: the bit i'm unsure about is what sequence of events
>> are required to happen.
>
> Not sure if this is the best answer or not, but you might enjoy
> looking at the git-remote-helpers(7) feature

       git-remote-helpers - Helper programs to interact with remote
       repositories

okaay, i see where you're going with that one.  thanks.

> and git-remote-testgit.py
> example.

 have to find that.

> Also, in case it is useful: the old scripted "git fetch" is available
> as contrib/examples/git-fetch.sh.

 ahh.  ah _ha_.  yes.  excellent.

 btw, gitpython interestingly has a 4-line method for working out the
list of commits required to be obtained in order for the local copy to
catch up:

        repo_refs = self.git.rev_list(ref, '--').strip().splitlines()
        other_repo_refs = other_repo.git.rev_list(other_ref,
'--').strip().splitlines()
        diff_refs = list(set(other_repo_refs) - set(repo_refs))
        return map(lambda ref: Commit.find_all(other_repo, ref,
max_count=1)[0], diff_refs)

it's going to be a leeetle more tricky than that, over a p2p network,
unless i cheat, and place the other_repo's rev-list into a meta-file
which is also distributed over the p2p network.... yuk to that, on the
basis that long-living repos that's going to be a biiig file (would
that actually be the case?  what's the largest "git rev-list --all"
that anyone's ever seen?  yikes - on git itself it's 860k!)   hmmm..
git rev {commit1}...{commit2} might give me what i need (and be
equivalent to the above 4 python lines but it can be done by the
remote end - that's still potentially a damn big amount of info.
well, i'm sure i'll cope :)

i just want to get something working, damnit.  there simply aren't
enough free software client-server-paradigm-independent development
tools around, and that's a situation that's causing me some concern.

l.

Re: git peer-to-peer project: info needed

[Matthieu Moy]

Luke Kenneth Casson Leighton <luke.leighton@gmail.com> writes:

> hi folks,
>
> [please could you kindly cc on responses because i am subscribed with
> "no mail" set]
>
> i need some guidance on what i should be doing, to add peer-to-peer
> networking to "git fetch".

There have already been some attempts at a "gittorrent" mechanism.
Google will tell you more about that.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: git peer-to-peer project: info needed

[Luke Kenneth Casson Leighton]

On Mon, Aug 30, 2010 at 7:56 AM, Matthieu Moy
<Matthieu.Moy@grenoble-inp.fr> wrote:
> Luke Kenneth Casson Leighton <luke.leighton@gmail.com> writes:
>
>> hi folks,
>>
>> [please could you kindly cc on responses because i am subscribed with
>> "no mail" set]
>>
>> i need some guidance on what i should be doing, to add peer-to-peer
>> networking to "git fetch".
>
> There have already been some attempts at a "gittorrent" mechanism.
> Google will tell you more about that.

 i know.

 it's best to assume that i've been following those, given that i
wrote the advogato and slashdot articles which brought sam and jonas'
efforts to peoples' attention, but for the sake of brevity in
contacting the git list i didn't want to mention that, so i apologise
for not mentioning that i'm aware of gittorrent.

 sam has already ruled out the bittorrent protocol as a means to
create "mirrorsync".  mirrorsync is, as it stands, a lower-level
protocol requiring the addition of DHT and other peer-to-peer
infrastucture (NAT-busting), and sam is designing mirrorsync to be
part of git-daemon (i.e. it requires an HTTP port).

i believe that the use of HTTP is a mistake, and i believe that a
proper peer to peer git distribution protocol _requires_
bittorrent-like features, in order to have a chance of success (i.e.
be "simple" enough to use i.e. _not_ require knowledge of firewall
configuration etc.)

 so whilst this is all way outside of the scope of the git mailing
list, i'm describing the rough plan here in case anyone's interested:
the rough plan is to create a VFS layer into which i can then work
"pack objects" into quotes torrents quotes, named by filename after
the SHA1 hash.  the bittorrent protocol is perfectly capable of
supporting multiple files; thus it should not be too hard a job to rip
out the hard-coded filesystem access in the bittornado source code -
os.listdir, open(fname, "r"/"w"), osstat etc - and then redirect the
file/directory operations onto underlying git operations.

l.

Re: git peer-to-peer project: info needed

[Casey Dahlin]

On Sun, Aug 29, 2010 at 10:56:27PM +0100, Luke Kenneth Casson Leighton wrote:
> hi folks,
> 
> [please could you kindly cc on responses because i am subscribed with
> "no mail" set]
> 
> i need some guidance on what i should be doing, to add peer-to-peer
> networking to "git fetch".  i can take care of the peer-to-peer
> networking side: the bit i'm unsure about is what sequence of events
> are required to happen.  i'm presently looking at the use of
> walker_fetch in builtin-http-fetch.c which gives me some idea.
> however as i aim to implement this first in python not c, i need to be
> using git via command-line only.
> 

I have a peer-to-peer git patchset that I'm already working on, which is
nearly in a showable state. I've wanted to sand off a few more edges
before a public review but I'll happily share it.

--CJD

Re: git peer-to-peer project: info needed

[Casey Dahlin]

On Mon, Aug 30, 2010 at 01:39:07PM -0400, Casey Dahlin wrote:
> On Sun, Aug 29, 2010 at 10:56:27PM +0100, Luke Kenneth Casson Leighton wrote:
> > hi folks,
> > 
> > [please could you kindly cc on responses because i am subscribed with
> > "no mail" set]
> > 
> > i need some guidance on what i should be doing, to add peer-to-peer
> > networking to "git fetch".  i can take care of the peer-to-peer
> > networking side: the bit i'm unsure about is what sequence of events
> > are required to happen.  i'm presently looking at the use of
> > walker_fetch in builtin-http-fetch.c which gives me some idea.
> > however as i aim to implement this first in python not c, i need to be
> > using git via command-line only.
> > 
> 
> I have a peer-to-peer git patchset that I'm already working on, which is
> nearly in a showable state. I've wanted to sand off a few more edges
> before a public review but I'll happily share it.
> 
> --CJD

On further review you seem to be talking more about making fetch itself
peer-to-peer, where my patchset is more of a peer-to-peer branch-sharing
mechanism. Complementary but not necessarily conflicting. Still I'll
hurry up and get it online :)

--CJD

Re: git peer-to-peer project: info needed

[Luke Kenneth Casson Leighton]

On Mon, Aug 30, 2010 at 6:45 PM, Casey Dahlin <cdahlin@redhat.com> wrote:

> On further review you seem to be talking more about making fetch itself
> peer-to-peer,

 only on the basis that that's what i believed would be a good start,
lacking any experience or advice :)

> where my patchset is more of a peer-to-peer branch-sharing
> mechanism.

 what's the difference?  i did look at making "blobs / trees"
available but i figured that that would just get you a snapshot (not
helpful).  i'm afraid i don't actually know the difference between
fetch sharing and branch sharing, as my knowledge of git internals is
a bit "swiss cheese".


> Complementary but not necessarily conflicting. Still I'll
> hurry up and get it online :)

 commented already.  was a bit of a bit message in the inbox, with an
inline thingy rather than attachment, but i'll cope.

l.