Gittorent .. avahi ?

Gittorent .. avahi ?

[Srijak Rijal]

Hi all,
I hope this is the right forum for this :).
 I hope to be able to implement the gittorrent client/ tracker during
GoogleSoc (if chosen :) ). I have been reading up on various git docs
and listserv messages to figure out what features I want in it in
addition to the basics. I have been concentrating especially on how to
minimise tracker load.
I was thinking about enabling avahi in the clients so that they can
find peers without putting that much extra load on the tracker.
A small caveat is that avahi timeout is around 50ms, so this sort of
dynamic peer discovery would probably turn out to work effectively
only among clients in the same LAN. However, best case scenario is
clients in the same LAN would have only a couple of clients talking to
the tracker and spreading the data(as well as peer list etc) in their
LAN.

To me, this feature seems be worth implementing if enough git users
are in the same LAN trying to get at a certain git repository.
What do you guys think ?

-Srijak

Re: Gittorent .. avahi ?

[Sam Vilain]

Srijak Rijal wrote:
> Hi all,
> I hope this is the right forum for this :).
>  I hope to be able to implement the gittorrent client/ tracker during
> GoogleSoc (if chosen :) ). I have been reading up on various git docs
> and listserv messages to figure out what features I want in it in
> addition to the basics. I have been concentrating especially on how to
> minimise tracker load.
>   

Note that a minimal tracker can always return the same static file,
listing known stable peers; the P2P protocol is capable of discovering
new peers.

Also, unlike BitTorrent, the protocol is capable of negotiating the
tracker update frequency.  A tracker under load would just start
advertising addresses for longer, and returning server busy responses
for peers that contact them too frequently.  See the "valid" key in the
Request, and the "expires" key in the Response.

I think in BitTorrent there is an *over*-emphasis on making the tracker
load light, because people want to be able to run them on the most
dirt-cheap hosting account they can find, because they might get shut
down and don't want a huge outlay.  But with GitTorrent, a large number
of the tracker providers are going to be software houses, and people
already running mirroring services - and so the increased load will be
more than justified by the reduced overall load.

> I was thinking about enabling avahi in the clients so that they can
> find peers without putting that much extra load on the tracker.
> A small caveat is that avahi timeout is around 50ms, so this sort of
> dynamic peer discovery would probably turn out to work effectively
> only among clients in the same LAN. However, best case scenario is
> clients in the same LAN would have only a couple of clients talking to
> the tracker and spreading the data(as well as peer list etc) in their
> LAN.
>
> To me, this feature seems be worth implementing if enough git users
> are in the same LAN trying to get at a certain git repository.
> What do you guys think ?
>   

Looks like avahi is only intended for the local network anyway, going by
the home page and Wikipedia article.

I guess the use cases for this include programming houses and hackathons.

At a hackathon, you might scan the network for a list of projects, and
jump into one of the swarms.  Then you hack on the code.  Then to
"commit", you set up and advertise another tracker that has your list of
heads, but refers to the other repo as an "alternative" (see
Metainfo/repo/alternatives).  The other people's avahi-enhanced
gittorrent client could see that a new related project has been
advertised, and fetch new commits to "remotes/" references automatically.

SVN fans would probably then configure it to automatically merge in new
remote references to their current working tree, but that's not
important right now :-).

One current TO-DO item in the specification to make this use case
scalable is to allow connections to deal with multiple (possibly
related) repositories at the same time.  Otherwise, with 6 "committers",
you've got 6 P2P swarms.  But this can happen with the next incremental
version of the specification.

At a programming house, setting up a tracker to commit wouldn't be as
frequent, as you expect people to have the appropriate access to
commit.  It would still be useful as a live directory of projects - but
is it significantly more useful than a simple gitweb?

Anyway, the old warning about scope creep applies to this idea - working
from the bottom up is a lot better for getting things done than thinking
big.  Just look at me, I think big all the time and never get anything
done ;-)  There will be a lot of interesting technologies enabled by
gittorrent.

Sam.

Re: Gittorent .. avahi ?

[Shawn O. Pearce]

Sam Vilain <sam@vilain.net> wrote:
...
> I think in BitTorrent there is an *over*-emphasis on making the tracker
> load light, because people want to be able to run them on the most
> dirt-cheap hosting account they can find, because they might get shut
> down and don't want a huge outlay.  But with GitTorrent, a large number
> of the tracker providers are going to be software houses, and people
> already running mirroring services - and so the increased load will be
> more than justified by the reduced overall load.
...

Yes, I agree entirely.  Almost anyone who is going to be running a
tracker for GitTorrent would already today be running the anonymous
Git daemon.  These users (ahm... kernel.org) would like to reduce
load on their servers if clients can get the data from elsewhere.
Hosting a tracker is still load, but should be a *much* lower load.

Note that kernel.org's entire load appears to be disk IO (as stated
by hpa on many occasions).  A little bit of extra CPU and network
bandwidth (for tracker maintenance) to cut disk IO in half (due to
less object directory accesses) would probably be major improvement
for them, and would still cut their overall network bandwidth as
clients would get packs from other nodes in the P2P system.

> Anyway, the old warning about scope creep applies to this idea - working
> from the bottom up is a lot better for getting things done than thinking
> big.  Just look at me, I think big all the time and never get anything
> done ;-)  There will be a lot of interesting technologies enabled by
> gittorrent.

I agree.  The biggest risk to any GSoC project is not the student,
or the community, or the technology... but its scope creep.
Of course this is true of *any* project.

Git evolved.  Linux evolved.  Apache evolved.  None of them set
out on day one with a huge list of "must have" features that they
coded from scratch.  Every one of those projects organically grew
as new features were desired and old bugs fixed.[*1*]


[*1*] Yes, I know, Linus did set out when writing both Linux and
      Git to create something that did X... but X was a pretty
      small concept.  Git started as a "track Linus' state thing",
      not as a full SCM.  ;-)

-- 
Shawn.

Re: Gittorent .. avahi ?

[Jonas Fonseca]

Hello Srijak,

On 4/1/07, Srijak Rijal <srijak@gmail.com> wrote:
>  I hope to be able to implement the gittorrent client/ tracker during
> GoogleSoc (if chosen :) ). I have been reading up on various git docs
> and listserv messages to figure out what features I want in it in
> addition to the basics. I have been concentrating especially on how to
> minimise tracker load.

While the tracker is a very interesting part of GitTorrent, because it
has a lot of responsibility for 'advertising' new changes, I think the
client, in itself, would be a hard and big enough project. And as Sam
said the tracker can be reduced to a simple static file that is
regularly updated by a cron job or something like that. On the other
hand, having a tracker might ease the task of implementing a client.
Hmm, chickens and eggs... ;)

I made a report about GitTorrent from which you might be able to
get some info from or at least some overall motivations and
considerations. However, it is for the much simpler and stupid
'initial' proposal, before Sam stepped in and made especially the
core of the peer wire protocol more scalable. So 'caveat emptor'
(this especially goes for the more Git related parts).

You can find it at:

  http://jonas.nitro.dk/tmp/foo/gittorrent.html/node38.html

and the whole report at

  http://jonas.nitro.dk/tmp/foo/gittorrent.pdf

I really hope your project gets approved.
-- 
Jonas Fonseca (fonseca@diku.dk)

Re: Gittorent .. avahi ?

[Srijak Rijal]

Hi Jonas,
>
> While the tracker is a very interesting part of GitTorrent, because it
> has a lot of responsibility for 'advertising' new changes, I think the
> client, in itself, would be a hard and big enough project. And as Sam
> said the tracker can be reduced to a simple static file that is
> regularly updated by a cron job or something like that.

Yup. For now,  I plan to adhere closely to the spec and get the initial 
version out :).
I agree with Shawn in that letting the tool organically evolve to
include new features and bug fixes as required is the best way to go.
The avahi idea is a product of me having time to think about the
project, but not being able to code :).

> I made a report about GitTorrent from which you might be able to
> get some info from or at least some overall motivations and
> considerations. However, it is for the much simpler and stupid
> 'initial' proposal, before Sam stepped in and made especially the
> core of the peer wire protocol more scalable. So 'caveat emptor'
> (this especially goes for the more Git related parts).
>
> You can find it at:
>
>  http://jonas.nitro.dk/tmp/foo/gittorrent.html/node38.html
>
> and the whole report at
>
>  http://jonas.nitro.dk/tmp/foo/gittorrent.pdf
>
Great :). Thanks for the link. I think the report will be helpful.  I
intend to look it over as soon as I get time. School in kinda hectic ATM.

> I really hope your project gets approved.
Thanks, me too :),




Srijak.

Re: Gittorent .. avahi ?

[Jakub Narebski]

[Cc: Srijak Rijal <srijak@gmail.com>, git@vger.kernel.org]

Srijak Rijal wrote:

> I hope to be able to implement the gittorrent client/ tracker during
> GoogleSoc (if chosen :) ).

I hope that you have not missed deadline.

> I was thinking about enabling avahi in the clients so that they can
> find peers without putting that much extra load on the tracker.

Avahi is Zeroconf implementation... wouldn't it be better to think
about implementing Kademlia?

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: Gittorent .. avahi ?

[Srijak Rijal]

Hi Jakub,
> I hope that you have not missed deadline.
>
>   
I have already applied :)
> Avahi is Zeroconf implementation... wouldn't it be better to think
> about implementing Kademlia?
>
>   
Kademlia is something to look at. But not anytime soon :).
The current-"est" and working plan is to strictly follow the spec for
functionality.
Then, according to community requirement/recommendation etc. features
will be added.


Srijak.

Re: Gittorent .. avahi ?

[Jakub Narebski]

Srijak Rijal wrote:
> Hi Jakub,
>> I hope that you have not missed deadline.
>>
> I have already applied :)

I hope then that you would be accepted.

-- 
Jakub Narebski
Poland