Question about possible git races

Question about possible git races

[Radoslaw Szkodzinski]

[-- Attachment #1: Type: text/plain, Size: 919 bytes --]

I'd like to write a multithreaded application using git, so I'd like to see if 
there are any races:

- push vs pull
One thread pushes to the repository while another is pulling from it at the 
same time. I should get the older commit.

- push vs push
Both threads push at the same time. What happens?
Any good way to merge those pushes?
(I have full access to both repos)

Possibly those two aren't fast-forward of each other.
I think one of the pushes should abort in this case unless I force it.

- fetch vs fetch
I mean that two threads try to fetch from different repositories to a single 
one. Possibly those two aren't fast-forward of each other.
Any good way to merge those fetches?
(I have full access to both repos)

I'm meaning really bare git there, w/o bash+perl scripts.

-- 
GPG Key id:  0xD1F10BA2
Fingerprint: 96E2 304A B9C4 949A 10A0  9105 9543 0453 D1F1 0BA2

AstralStorm

[-- Attachment #2: Type: application/pgp-signature, Size: 191 bytes --]

Re: Question about possible git races

[Radoslaw Szkodzinski]

[-- Attachment #1: Type: text/plain, Size: 538 bytes --]

On Monday 20 March 2006 17:24, Radoslaw Szkodzinski wrote yet:
> I'd like to write a multithreaded application using git, so I'd like to see
> if there are any races:
>
> - push vs fetch
> ...
> - push vs push
> ...
> - fetch vs fetch
> ...
>
> I'm meaning really bare git there, w/o bash+perl scripts.

Could anyone try to answer the question? 
I'd really like to know, because it's crucial to my application.

-- 
GPG Key id:  0xD1F10BA2
Fingerprint: 96E2 304A B9C4 949A 10A0  9105 9543 0453 D1F1 0BA2

AstralStorm

[-- Attachment #2: Type: application/pgp-signature, Size: 191 bytes --]

Re: Question about possible git races

[Andreas Ericsson]

Radoslaw Szkodzinski wrote:
> I'd like to write a multithreaded application using git,

Why on earth? The git tools aren't written to be thread-safe in that 
manner, so you'll run into all sorts of problems. Unless you're talking 
about managing the code for a multithreaded application with git, in 
which case you should just go read the tutorial. However, feeling 
slightly tipsy and in a distinctly good mood, I shall try to answer your 
questions anyway.


> so I'd like to see if 
> there are any races:
> 
> - push vs pull
> One thread pushes to the repository while another is pulling from it at the 
> same time. I should get the older commit.
> 

You will. Git atomizes (atomicizes? atomicifies?) pushes by updating the 
branch head being pushed to after all the commit-, tree- and 
blob-objects are written. Tags are handled separately but equally 
atomically.


> - push vs push
> Both threads push at the same time. What happens?
> Any good way to merge those pushes?
> (I have full access to both repos)
> 
> Possibly those two aren't fast-forward of each other.
> I think one of the pushes should abort in this case unless I force it.
> 

Read the source to find out if it's locking the repo while updating or 
not (I think it is, but I'm not sure). If it isn't the last one to 
finish pushing wins out since the branch head update from that push will 
overwrite the previous one.


> - fetch vs fetch
> I mean that two threads try to fetch from different repositories to a single 
> one. Possibly those two aren't fast-forward of each other.
> Any good way to merge those fetches?
> (I have full access to both repos)
> 

git help octopus

You can fetch those two remote branch heads to local branches 
simultaneously and then do the octopus in the master-thread while no 
other updates are happening. Doing several simultanous merges to a 
single branch is quite frankly so insane I have to go get myself a drink 
just from thinking about it.

> I'm meaning really bare git there, w/o bash+perl scripts.
> 

I don't think you can do it without Python. The default merge strategy 
is written in python, so.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: Question about possible git races

[Andreas Ericsson]

Radoslaw Szkodzinski wrote:
> On Monday 20 March 2006 17:24, Radoslaw Szkodzinski wrote yet:
> 
> Could anyone try to answer the question? 
> I'd really like to know, because it's crucial to my application.
> 

I believe the reasons no-one answered your first mail are, in the 
following order:

1. Since I'm sure you're truly capable of writing such an application 
and finishing it before the git API has changed completely you should 
have gotten your answers from trial-and-error, reading the source, or by 
just trying the app and seeing where it fails.

2. You didn't say *why* you want to write a multi-threaded layer on top 
of git. Is it to implement a redundant file-server with revision 
control? If so you don't need multi-threading. You need clever update 
hooks, a master repo and plenty of bandwidth with fast disks on the 
file-servers.

3. You didn't mention what you've tried to find the answers yourself, 
which makes me think you want me and the rest of us gitizens (yay! I 
coined a phrase) do your homework for you. I personally find it very 
rude that you send another email again so shortly after the first one 
claiming that "you need this info for your app", when the 500-odd people 
you're asking clearly need their time for their families, hobbies, 
daytime jobs, beer, etc. etc...

4. Noone felt like answering since they saw no use for a multi-threaded 
layer on top of git, especially without knowing what it was for. 
Friendliness only goes so far when met by such lack of respect for other 
peoples time, but if someone had seen the uses for the app you're 
writing they probably would have taken time to at least ask you for some 
of the answers you left out in your original mail. If nothing else for 
the sake of curiosity.


Some more pointers on how to get answers to questions posted in online 
forums can be found on the links below.

http://catb.org/~esr/faqs/smart-questions.html
http://www.catb.org/~esr/faqs/hacker-howto.html


Btw. I'm assuming you're aware you'll have to GPL this app of yours, 
since git is GPL and you'll be using the git produce in a way that makes 
it vital to your app.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: Question about possible git races

[Junio C Hamano]

Radoslaw Szkodzinski <astralstorm@o2.pl> writes:

> - push vs pull
>
> - push vs push
>
> - fetch vs fetch

About push vs push, with "really bare git", I take it that you
mean two send-pack from remote sites running two receive-pack
simultaneously.

There is an explicit race avoidance between the receive-pack
processes.  When a ref (either branch head or a tag) is updated,
it does:

 - read the current value from the ref.
 - do its work.
 - lock to prevent others to create the temporary file for
   updating the ref.
 - create the temporary file for the ref and write the new value.
 - check if the ref's value has not changed from what it
   initially read;
 - rename the temporary file to the ref to unlock.

Read receive-pack.c::update() for exact details if you are
interested.

> I'm meaning really bare git there, w/o bash+perl scripts.

The question does not make any sense for other cases, because
branch update by fetch and pull are all scripts based.

Re: Question about possible git races

[Radoslaw Szkodzinski]

[-- Attachment #1: Type: text/plain, Size: 1681 bytes --]

On Thursday 23 March 2006 01:24, Junio C Hamano wrote yet:
> Radoslaw Szkodzinski <astralstorm@o2.pl> writes:
> > - push vs pull
> >
> > - push vs push
> >
> > - fetch vs fetch
>
> About push vs push, with "really bare git", I take it that you
> mean two send-pack from remote sites running two receive-pack
> simultaneously.
>
> There is an explicit race avoidance between the receive-pack
> processes.  When a ref (either branch head or a tag) is updated,
> it does:
>
>  - read the current value from the ref.
>  - do its work.
>  - lock to prevent others to create the temporary file for
>    updating the ref.


>  - create the temporary file for the ref and write the new value.
>  - check if the ref's value has not changed from what it
>    initially read;
>  - rename the temporary file to the ref to unlock.
>
> Read receive-pack.c::update() for exact details if you are
> interested.

So there is locking I've missed while reading through the source.
Guess all the coffee doesn't help.

There is a lock, so the other git-receive-pack for given ref will fail. 
Does that also work for git-local-fetch with -l option?

Looks good though that I can fetch to another ref.

>
> > I'm meaning really bare git there, w/o bash+perl scripts.
>
> The question does not make any sense for other cases, because
> branch update by fetch and pull are all scripts based.

I should have known better than to use vague words.
For me fetch = git-*-fetch. Which in turn calls git-receive-pack.

Thank you for the precise answer.

-- 
GPG Key id:  0xD1F10BA2
Fingerprint: 96E2 304A B9C4 949A 10A0  9105 9543 0453 D1F1 0BA2

AstralStorm

[-- Attachment #2: Type: application/pgp-signature, Size: 191 bytes --]

Re: Question about possible git races

[Radoslaw Szkodzinski]

[-- Attachment #1: Type: text/plain, Size: 3120 bytes --]

On Thursday 23 March 2006 00:28, Andreas Ericsson wrote yet:
> However, feeling  slightly tipsy and in a distinctly good mood, I shall try 
to answer your questions anyway.

The golden liquid works for me, great. Thank you.

>
> > so I'd like to see if
> > there are any races:
> >
> > - push vs pull
> > One thread pushes to the repository while another is pulling from it at
> > the same time. I should get the older commit.
>
> You will. Git atomizes (atomicizes? atomicifies?) pushes by updating the
> branch head being pushed to after all the commit-, tree- and
> blob-objects are written. 

Just as I expected. Good.

> Tags are handled separately but equally 
> atomically.

Good too.

>
> > - fetch vs fetch
> > I mean that two threads try to fetch from different repositories to a
> > single one. Possibly those two aren't fast-forward of each other.
> > Any good way to merge those fetches?
> > (I have full access to both repos)
>
> git help octopus
>
> You can fetch those two remote branch heads to local branches
> simultaneously and then do the octopus in the master-thread while no
> other updates are happening.

I could slurp the unrelated commits with an octopus of course...
But the others pose a problem.

> Doing several simultanous merges to a 
> single branch is quite frankly so insane I have to go get myself a drink
> just from thinking about it.

It's a rare situation in the app, but not impossible. (I want to avoid 
locking) That's why I was asking about it.

>
> > I'm meaning really bare git there, w/o bash+perl scripts.
>
> I don't think you can do it without Python. The default merge strategy
> is written in python, so.

You've hit it, the app is written in Python (as of yet).

The default merge strategy is simply... calling merge and also detecting 
naming/existence conflicts with a simple scalar merge.

The only bit more complicated thing is detecting the LCA for 3-way merge.


I'd like to build a decentralised collaborative web application, as scalable 
as it gets. (I expect a lot of traffic)
I also need to only check out parts of the tree. (many SCMs can't do that)
Git, with its speed, seems well-suited to the task.
Merging will be uncommon in the workload, so can be slow, but shouldn't break 
or present weird conflicts to the users.

Unless the accidental case starts to dominate - then I'll have a problem.

Sorry for spam and cutting out major questions, then answering at the end of 
the post.


> Btw. I'm assuming you're aware you'll have to GPL this app of yours,
> since git is GPL and you'll be using the git produce in a way that makes
> it vital to your app.

It will be, but not because of git (it's execve()ing it), but rather because 
of the principle.

Intermediate results will probably be:
- a good Python object interface to git
- another porcelain, portable, coded in Python
- maybe a new merge strategy
- later yet another porcelain, written in C

-- 
GPG Key id:  0xD1F10BA2
Fingerprint: 96E2 304A B9C4 949A 10A0  9105 9543 0453 D1F1 0BA2

AstralStorm

[-- Attachment #2: Type: application/pgp-signature, Size: 191 bytes --]

Re: Question about possible git races

[Junio C Hamano]

Radoslaw Szkodzinski <astralstorm@o2.pl> writes:

> For me fetch = git-*-fetch. Which in turn calls git-receive-pack.

Does anything other than git-send-pack call git-receive-pack?

For fetch, git-fetch-pack is called from the core level, but it
does not update refs itself.  It writes out enough information
to its standard output so that the script calling it can update
the refs.  So at the core level there cannot be any race, but
that does not necessarily mean existing scripts are race free.

Our barebone Porcelainish scripts _do_ use update-ref to do the
same lock - re-read - rename-to-update cycle when updating the
refs using that information, but that is something you
explicitly said you are not interested in ;-).

Re: Question about possible git races

[Andreas Ericsson]

Radoslaw Szkodzinski wrote:
> On Thursday 23 March 2006 00:28, Andreas Ericsson wrote yet:
> 
>>Btw. I'm assuming you're aware you'll have to GPL this app of yours,
>>since git is GPL and you'll be using the git produce in a way that makes
>>it vital to your app.
> 
> 
> It will be, but not because of git (it's execve()ing it), but rather because 
> of the principle.
> 

Ah, Good Thing. Just to clarify for the archives though, it *is* 
required since it's using git in a way that makes git a fundamental, 
non-replaceable part of its core operations.

> Intermediate results will probably be:
> - later yet another porcelain, written in C
> 

yagit? yagp? What I wanna know is when jigit's gonna hit the streets. :)



-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Re: Question about possible git races

[Radoslaw Szkodzinski]

[-- Attachment #1: Type: text/plain, Size: 1404 bytes --]

On Thursday 23 March 2006 03:55, Andreas Ericsson wrote yet:
> Radoslaw Szkodzinski wrote:
> > On Thursday 23 March 2006 00:28, Andreas Ericsson wrote yet:
> >>Btw. I'm assuming you're aware you'll have to GPL this app of yours,
> >>since git is GPL and you'll be using the git produce in a way that makes
> >>it vital to your app.
> >
> > It will be, but not because of git (it's execve()ing it), but rather
> > because of the principle.
>
> Ah, Good Thing. Just to clarify for the archives though, it *is*
> required since it's using git in a way that makes git a fundamental,
> non-replaceable part of its core operations.
>
> > Intermediate results will probably be:
> > - later yet another porcelain, written in C
>
> yagit? yagp? What I wanna know is when jigit's gonna hit the streets. :)

I need a memorable three-letter acronym, so it will be gip. (Git porcelain In 
Python) and the C one... later.

GIP also stands for Good Informatics Practices :P or "to swindle", but it's 
the deprecated usage, I think. It has some racial ties.

Conflicts with: GNOME IP calculator - http://www.debain.org/software/gip/

I'd get myself a better commandline IP calculator than give a good TLA to 
this. Gnip should be the name.

What's jigit? Did I miss something?

-- 
GPG Key id:  0xD1F10BA2
Fingerprint: 96E2 304A B9C4 949A 10A0  9105 9543 0453 D1F1 0BA2

AstralStorm

[-- Attachment #2: Type: application/pgp-signature, Size: 191 bytes --]