http://tech.slashdot.org/comments.pl?sid=1885890&cid=34358134

http://tech.slashdot.org/comments.pl?sid=1885890&cid=34358134

[Luke Kenneth Casson Leighton]

On Wed, Sep 22, 2010 at 11:20 AM, Luke Kenneth Casson Leighton
<luke.leighton@gmail.com> wrote:
> i trust that nobody will make the comment "but source code couldn't
> possibly be considered dangerous enough to block using ACTA" and this
> might go some way towards explaining one - not all - one i repeat just
> one of the factors why i believe that free software development
> _needs_ tools which can side-step single-point-of-failure locations
> that can be blocked by stupid fucking "agreements" like ACTA.

 i trust that the above subject-line comment, or more specifically its
constituent links, begins to hint at why i believe it is important
that git-over-p2p be given a higher priority than it is at present.
you can easily join the dots to see where this government insanity is
going: it won't be long before this insanity is utilised to take
control of free software web sites just because they happen to be
hosting tools which MIGHT be utilised "for copyright infringment".
it's bad enough that the DMCA exists and can be used to intimidate
sourceforget into removing rtmpdump - now imagine if someone decided
to take control of sourceforget ok actually yeah maybe that's not so
great a loss after all but it's the PRINCIPLE damnit :)

 l.

Re: http://tech.slashdot.org/comments.pl?sid=1885890&cid=34358134

[Ævar Arnfjörð Bjarmason]

On Sat, Nov 27, 2010 at 18:33, Luke Kenneth Casson Leighton
<luke.leighton@gmail.com> wrote:

> I believe it is important that git-over-p2p be given a higher
> priority than it is at present.

We give "priority" to the stuff people submit patches for. There isn't
a git-over-p2p because nobody is working on it, would you like to work
on it?

I'm also not convinced that this is actually needed. It's trivial to
set up a p2p-like network by just emulating a darknet by manually
adding remotes & fetching/pushing.

That's not viable for the stuff that usually gets distributed on p2p
networks, but it sure is viable when you have people working on the
same codebase. Which would be the use case for a git-over-p2p client,
right?

Re: http://tech.slashdot.org/comments.pl?sid=1885890&cid=34358134

[Luke Kenneth Casson Leighton]

On Sat, Nov 27, 2010 at 6:36 PM, Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> On Sat, Nov 27, 2010 at 18:33, Luke Kenneth Casson Leighton
> <luke.leighton@gmail.com> wrote:
>
>> I believe it is important that git-over-p2p be given a higher
>> priority than it is at present.
>
> We give "priority" to the stuff people submit patches for. There isn't
> a git-over-p2p because nobody is working on it, would you like to work
> on it?

 i was / have been.  unfortunately it's unfunded work, and, thanks to
receiving less money than i require for living and for feeding my
family i now face a court hearing on 16th december (3 weeks time)
which has the aim and ultimate goal of making us homeless.  i
therefore respectfull request that you please don't try to make me
feel like i _should_ be working on this in order to "earn your
respect" ( that goes for you too, johnathon ).

> I'm also not convinced that this is actually needed. It's trivial to
> set up a p2p-like network by just emulating a darknet by manually
> adding remotes & fetching/pushing.

 ah.. but that's the single-use case for a single shared repository.
git "as-is" the fetch/push over http is limited to a single
repository.  i'm thinking more in terms of a global network, with
searches for checksummed objects "accidentally" potentially coming
from wildly different repositories.

 the work i did a few months ago showed that just "dropping" git on
top of e.g. the bittorrent protocol as-is, files/dirs mapped to git
objects, wouldn't cut the mustard, because the bloody bittorrent
protocol "chunks" system treats the entire fileset as an ordered but
contiguous "stream" of data (metadata in the actual .torrent file says
how long each of the files are and in what order they are, and this is
utilised at the receiving end to reconstruct the files after the
datablocks have been received.  it means that if you want just one
single file, even of 2 bytes in length, you might have to pull 2
blocks of e.g. 256k and grab the byte and the end of the 1st and the
byte at the beginning of the 2nd! yukk...)

but i believe that a system where each git object was given its own
.torrent file _would_ work, and you utilise DHT or other mechanism
(perhaps even a torrent containing the metadata!) to get the
structural information.  the success of this model depends on everyone
being connected to everyone (a global network), in order to be able to
do a DHT search for the node(s) that are seeding that git object's
torrent file.

 the "global network" concept has the advantage of being far more
robust in the face of DNS take-overs, as you actually just need to
know _one_ DNS name out of potentially an unlimited number of DNS
names (or even an IP address or set of IP addresses) and you're joined
to other peers.

> That's not viable for the stuff that usually gets distributed on p2p
> networks, but it sure is viable when you have people working on the
> same codebase.

 codebase-in-a-git, wiki-in-a-git, website-in-a-git,
backups-of-filesystem-in-a-git (joey hess, one of the most amazing
debian developers around, has done all these things :)

> Which would be the use case for a git-over-p2p client,
> right?

yes it would... but i'm thinking well beyond that :)

at first glance the concept of git-over-darknet for example looks
great.  however it doesn't satisfy what i believe to be one of the key
strategic requirements to be "many-pushers to many-pullers real-time
resilient".  lose access to the darknet, or even if the git server
dies, what do you do??  if it's a darknet, you've got serious
problems: you can't find out who it was!

git-over-p2p on the other hand _would_ be [m-p-t-m-p-r-t-r], because
once the git-objects have been pulled by one client, they can be
pulled from that client by other clients too.  it's this feature of
p2p distribution that is so distinct from the way in which git is
normally described as "a distributed revision control system", and
makes it a bugger to explain what the fuss is all about over this
git-p2p stuff :)

 l.

Re: http://tech.slashdot.org/comments.pl?sid=1885890&cid=34358134

[Jonathan Nieder]

Luke Kenneth Casson Leighton wrote:

>                                                             i
> therefore respectfull request that you please don't try to make me
> feel like i _should_ be working on this in order to "earn your
> respect" ( that goes for you too, johnathon ).

Just for clarity: I never meant to imply such a thing.  Just that
messages like the one you just sent have quite an off-topic feeling
and that this mailing list is better for design advice, usage advice,
bug reporting, patch review, and similar things.  Just my own
opinion.

Kind regards,
Jonathan

Re: http://tech.slashdot.org/comments.pl?sid=1885890&cid=34358134

[Ævar Arnfjörð Bjarmason]

On Sat, Nov 27, 2010 at 20:06, Luke Kenneth Casson Leighton
<luke.leighton@gmail.com> wrote:
> On Sat, Nov 27, 2010 at 6:36 PM, Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>> On Sat, Nov 27, 2010 at 18:33, Luke Kenneth Casson Leighton
>> <luke.leighton@gmail.com> wrote:
>>
>>> I believe it is important that git-over-p2p be given a higher
>>> priority than it is at present.
>>
>> We give "priority" to the stuff people submit patches for. There isn't
>> a git-over-p2p because nobody is working on it, would you like to work
>> on it?
>
>  i was / have been.  unfortunately it's unfunded work, and, thanks to
> receiving less money than i require for living and for feeding my
> family i now face a court hearing on 16th december (3 weeks time)
> which has the aim and ultimate goal of making us homeless.  i
> therefore respectfull request that you please don't try to make me
> feel like i _should_ be working on this in order to "earn your
> respect" ( that goes for you too, johnathon ).

I don't mean to discourage you. But as Jonathan points out this list
is much better suited for submitting patches and discussing technical
details.

I'd be happy to review your patches, I'm very interested in getting
something like this myself. If you've been working on something I'd
love to see it.

And I hope you'll do all-right in your personal life.

>> I'm also not convinced that this is actually needed. It's trivial to
>> set up a p2p-like network by just emulating a darknet by manually
>> adding remotes & fetching/pushing.
>
>  ah.. but that's the single-use case for a single shared repository.
> git "as-is" the fetch/push over http is limited to a single
> repository.  i'm thinking more in terms of a global network, with
> searches for checksummed objects "accidentally" potentially coming
> from wildly different repositories.

Right, but is there an actual use case for people who are developing
code to use something like git over p2p? Maybe I'm just being
unimaginative, but I can't see a case where people are working on the
same project and can't find a way to push/pull from each other using
the existing methods. Especially since it's easy to sign up for free
Git hosting and use something like Tor to pull/push from there. Or to
set up your own git HTTP server on a Tor *.onion server.

But of course using the Git data format for non-source-code would be
very interesting, and much better for something like git-p2p.

Anyway, sorry about the brief reply. I'm not really knowledgable about
this, and didn't fully grok/read your reply (going out).

Re: http://tech.slashdot.org/comments.pl?sid=1885890&cid=34358134

[Will Palmer]

On Sat, 2010-11-27 at 21:19 +0100, Ævar Arnfjörð Bjarmason wrote:
> Right, but is there an actual use case for people who are developing
> code to use something like git over p2p? Maybe I'm just being
> unimaginative, but I can't see a case where people are working on the
> same project and can't find a way to push/pull from each other using
> the existing methods. Especially since it's easy to sign up for free
> Git hosting and use something like Tor to pull/push from there. Or to
> set up your own git HTTP server on a Tor *.onion server.

To me, the use-case wouldn't be because I /can't/ use existing methods,
it's because I /don't want to/ use existing methods :)
p2p tends to imply:
 - Resume-able downloads
 - Downloads from multiple sources at the same time
 - Transparently using an alternative source if one becomes unavailable
 - Load-balancing
 - referencing/linking based on "what it is" instead of "where it is"

The reason this stuff isn't a big itch for anybody is that these
problems tend to come about when cloning, which doesn't really happen
that often. When there's a new release, fetching the latest
release-tag /might/ bring about the same needs, though in my experience
"there's a new release" generally means nothing to me, since I check if
there's a new release by running "git remote update" and looking for new
tags, merging with my local patches if I see a relevant one.

I'm not convinced that any of this is something which is git's job, so
much as it sounds like something which could benefit from another
transport layer that git can optionally tie into. Sure, these could be
developed together (p2p with git in mind), but adding a giant p2p
section to the git codebase sounds like bloat.

I want my version control software to use p2p concepts for efficiency. I
don't want my version control software to be a p2p client any more than
I want my text-editor to be a mail client.

(In case it's not clear, I am in favour of p2p ("gittorrent")-like
functionality being in git)

Re: http://tech.slashdot.org/comments.pl?sid=1885890&cid=34358134

[Nguyen Thai Ngoc Duy]

On Mon, Nov 29, 2010 at 5:12 PM, Will Palmer <wmpalmer@gmail.com> wrote:
> On Sat, 2010-11-27 at 21:19 +0100, Ævar Arnfjörð Bjarmason wrote:
>> Right, but is there an actual use case for people who are developing
>> code to use something like git over p2p? Maybe I'm just being
>> unimaginative, but I can't see a case where people are working on the
>> same project and can't find a way to push/pull from each other using
>> the existing methods. Especially since it's easy to sign up for free
>> Git hosting and use something like Tor to pull/push from there. Or to
>> set up your own git HTTP server on a Tor *.onion server.
>
> To me, the use-case wouldn't be because I /can't/ use existing methods,
> it's because I /don't want to/ use existing methods :)
> p2p tends to imply:
>  ..
>  - Downloads from multiple sources at the same time

I would add "automatically" and it's a real case for me. I work on the
same project on different machines (each of them has different
resources that I need from time to time). So I clone to all the
machines. If I make a fix in one machine, other machines should
automatically fetch it. It's like a private p2p network with a single
user. If I add another machine to the network, all nodes should be
aware of it, without me doing "git remote add" on each node.
-- 
Duy

Re: http://tech.slashdot.org/comments.pl?sid=1885890&cid=34358134

[J.H.]

> I want my version control software to use p2p concepts for efficiency. I
> don't want my version control software to be a p2p client any more than
> I want my text-editor to be a mail client.

Keep in mind that adding p2p concepts to something doesn't make it more
efficient, in fact in most cases it makes it dramatically *LESS* efficient.

git-torrent like concepts have come up in the past, and I keep pointing
out how and why they likely won't be useful.  The biggest reason: there
is no advantage for a client to stay in the cloud once they have their
data.  You can force this, sure, but clones are seldom and rare to begin
with (as you mentioned) so there won't be a very large cloud to pull
from to start with.  With that in mind, I really see no advantage to p2p
inside of the git core at all.  It adds a lot of complexity for little gain.

Now you do mention things that would be useful:

- Ability to resume a clone that you only have a partial download for
(maybe just pack files?)
- Ability to include something like a 'meta-link' like list of
repositories to check for data (inferred from the multiple download
locations)

There are things we can learn from p2p, but I don't think adding it to
git is actually useful.

Just my $0.02 though.

- John 'Warthog9' Hawley

Re: http://tech.slashdot.org/comments.pl?sid=1885890&cid=34358134

[Will Palmer]

On Mon, 2010-11-29 at 11:27 -0800, J.H. wrote:
> > I want my version control software to use p2p concepts for efficiency. I
> > don't want my version control software to be a p2p client any more than
> > I want my text-editor to be a mail client.
> 
> Keep in mind that adding p2p concepts to something doesn't make it more
> efficient, in fact in most cases it makes it dramatically *LESS* efficient.

As a simple use-case:
Everyone comes into an office in the morning and runs "git remote
update". This potentially causes a lot of traffic between the office and
their offsite central repository. If this were a p2p scenario, the
transfer from the offsite could potentially happen only once.

That counts as "more efficient", to me.

> 
> git-torrent like concepts have come up in the past, and I keep pointing
> out how and why they likely won't be useful.  The biggest reason: there
> is no advantage for a client to stay in the cloud once they have their
> data.  

This is true of bittorrent as well: People stay in the cloud for
altruistic reasons.

You're thinking p2p in terms of "every peer serves data, and keeps
serving data. The network is more robust over time". I'm thinking p2p in
terms of "Every peer has the ability to serve data. Adding a server is
as trivial as adding a client."

The greatest advantage I can think of is "no existing server needs to
agree to the addition of a new server", or at least "the addition of an
existing server is accepted by convention, no questions asked"

> ..... You can force this, sure, but clones are seldom and rare to begin
> with (as you mentioned) so there won't be a very large cloud to pull
> from to start with.  With that in mind, I really see no advantage to p2p
> inside of the git core at all.  It adds a lot of complexity for little gain.

I agree that git itself is not a good place to explore p2p concepts. I
assume it would be much more useful to develop an independent p2p layer
and allow git to somehow use that.
And while there won't be "a large cloud", it's almost a guarantee that
there would be more than the /one/ server currently available during
clones or long fetches.

> Now you do mention things that would be useful:
> 
> - Ability to resume a clone that you only have a partial download for
> (maybe just pack files?)
> - Ability to include something like a 'meta-link' like list of
> repositories to check for data (inferred from the multiple download
> locations)
> 
> There are things we can learn from p2p, but I don't think adding it to
> git is actually useful.
> 
> Just my $0.02 though.
> 
> - John 'Warthog9' Hawley
> 

The biggest hurdle, I assume, would be "get git to talk to more than one
source of data at once", even if one needs to set those up manually. If
I understand correctly, packfiles are generated in a way which would not
necessarily be consistent between clones.

Re: http://tech.slashdot.org/comments.pl?sid=1885890&cid=34358134

[Jan Krüger]

--- Will Palmer <wmpalmer@gmail.com> wrote:

> As a simple use-case:
> Everyone comes into an office in the morning and runs "git remote
> update". This potentially causes a lot of traffic between the office
> and their offsite central repository. If this were a p2p scenario, the
> transfer from the offsite could potentially happen only once.

The same savings can be achieved by:

- Hosting the repository locally in the office;
- Automatically updating the 'official' location whenever something is
  pushed to the local repository;
- Having all developers use the office repository as their remote.

Takes about five minutes to set up.

(That does not invalidate your overall arguments, of course, but I
don't have time to address them in much detail.)

-Jan