concurrent access to multiple local git repos is error prone

concurrent access to multiple local git repos is error prone

[Alexander Mills]

If I have multiple local git repos, and I run `git status -v` on them,
sometimes I don't get any stdout for the command. This is highly
reproducible.

I assume that access more than 1 git repo concurrently on a local
machine is not without errors. However this seems like a huge
limitation or design flaw.

Is my observation correct? Are there any plans to remove this limitation?

My use case - I create a lot of developer tools and more than once I
have hit this limitation...I have to create a queue with concurrency
of 1 to run commands on git repos. It's very strange and
counterintuitive to have to do this.


-- 
Alexander D. Mills
¡¡¡ New cell phone number: (415)730-1805 !!!
alexander.d.mills@gmail.com

www.linkedin.com/pub/alexander-mills/b/7a5/418/

Re: concurrent access to multiple local git repos is error prone

[Alexander Mills]

Note, in my first paragraph, I should have said "If I have multiple
local git repos, and I run `git status -v` on them *concurrently*"...

-alex

On Sat, Aug 4, 2018 at 2:47 PM, Alexander Mills
<alexander.d.mills@gmail.com> wrote:
> If I have multiple local git repos, and I run `git status -v` on them,
> sometimes I don't get any stdout for the command. This is highly
> reproducible.
>
> I assume that access more than 1 git repo concurrently on a local
> machine is not without errors. However this seems like a huge
> limitation or design flaw.
>
> Is my observation correct? Are there any plans to remove this limitation?
>
> My use case - I create a lot of developer tools and more than once I
> have hit this limitation...I have to create a queue with concurrency
> of 1 to run commands on git repos. It's very strange and
> counterintuitive to have to do this.
>
>
> --
> Alexander D. Mills
> ¡¡¡ New cell phone number: (415)730-1805 !!!
> alexander.d.mills@gmail.com
>
> www.linkedin.com/pub/alexander-mills/b/7a5/418/



-- 
Alexander D. Mills
¡¡¡ New cell phone number: (415)730-1805 !!!
alexander.d.mills@gmail.com

www.linkedin.com/pub/alexander-mills/b/7a5/418/

Re: concurrent access to multiple local git repos is error prone

[Jonathan Nieder]

Hi Alexander,

Alexander Mills wrote:
> On Sat, Aug 4, 2018 at 2:47 PM, Alexander Mills
> <alexander.d.mills@gmail.com> wrote:

>> I assume that access more than 1 git repo concurrently on a local
>> machine is not without errors. However this seems like a huge
>> limitation or design flaw.
>>
>> Is my observation correct? Are there any plans to remove this limitation?
[...]
> Note, in my first paragraph, I should have said "If I have multiple
> local git repos, and I run `git status -v` on them *concurrently*"...

There's no known limitation of this kind, no.  It sounds quite
strange, and I've never heard of anything like that before.  Do you
have more details?

Thanks and hope that helps,
Jonathan

Re: concurrent access to multiple local git repos is error prone

[Jonathan Nieder]

Hi,

Alexander Mills wrote:

> Yeah as long as git shouldnt fumble wrt concurrent access across repos, it
> was most likely the same repo being accessed concurrently and that's what
> was causing the issue.
>
> that being said, it would be really nice if git itself could handle
> concurrent requests to the same repo, that definitely seems to be a
> limitation.

Instead of all these guesses, I'd prefer if you describe the actual
symptoms.

Git is meant to support concurrent accesses of the same repo as well,
and I haven't heard of any rumors of that being broken, either.

Sincerely,
Jonathan

Re: concurrent access to multiple local git repos is error prone

[Jonathan Nieder]

Alexander Mills wrote:

> Yeah in this case what appeared to be happening is that if `git status` was
> called concurrently, frequently I wouldnt get any stdout..

Thanks for reporting.  A few questions:

What platform are you on?  What version of Git are you running?  What
is the output of "git version --build-options"?

What is the exit status from these "git status" invocations that
didn't write output?  What options are you passing to "git status"?
Are there any other interesting symptoms?

Is this part of a larger tool or script?  Are there other operations
going on (e.g. something sending signals to these git processes)?

Does the repository pass "git fsck"?  Any other symptoms or hints that
could help in tracking this down?

Can you paste a transcript of the commands or script you ran and what
output it produced?  Is this something I should be able to reproduce?

Thanks and hope that helps,
Jonathan

Re: concurrent access to multiple local git repos is error prone

[Alexander Mills]

I am on Ubuntu, I will run the checks that you mentioned. I assume
it's operator error for now though. Like I said, it was just `git
status` that was sometimes failing to send any stdout, presumably when
access to a certain git repo was concurrent, but possibly due to
concurrent access across multiple repos.

Also, as an aside, this seems to be a bug, but probably a known bug:

$ git diff-index  HEAD; echo $?

:100755 100755 60e5d683c1eb3e61381b1a8ec2db822b94b9faec
0000000000000000000000000000000000000000 M      cli/npp_check_merge.sh
:100644 100644 35a453544de41e2227ab0afab31a396d299139e9
0000000000000000000000000000000000000000 M      src/find-projects.ts
:100644 100644 c1ee7bc18e6604cbf0d16653e9366109d6ac2ec9
0000000000000000000000000000000000000000 M      src/tables.ts
:100644 100644 29d9674fbb48f223f3434179d666b2aa991ad05a
0000000000000000000000000000000000000000 M
src/vcs-helpers/git-helpers.ts
0

$ git diff-index --quiet HEAD; echo $?
1

different exit codes depending on whether --quiet was used. In this
case, the exit code should be consistent.
The bug is with the `git diff-index` command, as you can see.

-alex

On Sun, Aug 5, 2018 at 1:11 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Alexander Mills wrote:
>
>> Yeah in this case what appeared to be happening is that if `git status` was
>> called concurrently, frequently I wouldnt get any stdout..
>
> Thanks for reporting.  A few questions:
>
> What platform are you on?  What version of Git are you running?  What
> is the output of "git version --build-options"?
>
> What is the exit status from these "git status" invocations that
> didn't write output?  What options are you passing to "git status"?
> Are there any other interesting symptoms?
>
> Is this part of a larger tool or script?  Are there other operations
> going on (e.g. something sending signals to these git processes)?
>
> Does the repository pass "git fsck"?  Any other symptoms or hints that
> could help in tracking this down?
>
> Can you paste a transcript of the commands or script you ran and what
> output it produced?  Is this something I should be able to reproduce?
>
> Thanks and hope that helps,
> Jonathan



-- 
Alexander D. Mills
¡¡¡ New cell phone number: (415)730-1805 !!!
alexander.d.mills@gmail.com

www.linkedin.com/pub/alexander-mills/b/7a5/418/

exit code in git diff-index [was: Re: concurrent access to multiple local git repos is error prone]

[Thomas Gummerer]

On 08/05, Alexander Mills wrote:
> Also, as an aside, this seems to be a bug, but probably a known bug:
> 
> $ git diff-index  HEAD; echo $?
> 
> :100755 100755 60e5d683c1eb3e61381b1a8ec2db822b94b9faec
> 0000000000000000000000000000000000000000 M      cli/npp_check_merge.sh
> :100644 100644 35a453544de41e2227ab0afab31a396d299139e9
> 0000000000000000000000000000000000000000 M      src/find-projects.ts
> :100644 100644 c1ee7bc18e6604cbf0d16653e9366109d6ac2ec9
> 0000000000000000000000000000000000000000 M      src/tables.ts
> :100644 100644 29d9674fbb48f223f3434179d666b2aa991ad05a
> 0000000000000000000000000000000000000000 M
> src/vcs-helpers/git-helpers.ts
> 0
> 
> $ git diff-index --quiet HEAD; echo $?
> 1
> 
> different exit codes depending on whether --quiet was used. In this
> case, the exit code should be consistent.
> The bug is with the `git diff-index` command, as you can see.

This is not a bug. 'git diff-index' (and 'git diff') only give an exit
code other than 0 in the default case if something actually goes wrong
with generating the diff, which in the usual case it shouldn't.

To get an exit code from 'git diff-index' if there are differences,
you'd have to pass the '--exit-code' flag.  The '--quite' flag implies
'--exit-code', as there's not much use in 'git diff --quiet' if
there's not even an exit code showing whether there are differences or
not.

The original patch (and more importantly the reasoning why
'--exit-code' is not the default behaviour for 'git diff') can be
found at [1].

[1]: https://public-inbox.org/git/81b0412b0703131717k7106ee1cg964628f0bda2c83e@mail.gmail.com/

> -alex

Re: concurrent access to multiple local git repos is error prone

[Alexander Mills]

Hi Johnathan,

Yeah this concurrency problem is real. Not only does it happen with
`git status` the same thing happens with `git rev-parse
--show-toplevel`.
What happens is that I get no stdout when repos are accessed
concurrently (and no stderr). If I limit concurrency to 1, the problem
goes away. When I up the concurrency, the problem is sporadic, which
is the exact signal for a concurrency/race-condition related issue.
The signs are damn clear. I have seen this problem on MacOS I think a
year back on a different project, but I never reported it b/c I hadn't
really verified it.

Like I said I am on Ubuntu. I have 3 git repos that are incorporated
into the tool that's generating the problem. For one repo I got this:

$ git fsck

Checking object directories: 100% (256/256), done.
dangling tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904

For `$ git version --build-options` I have:

git version 2.17.1
cpu: x86_64
no commit associated with this build
sizeof-long: 8


-alex



On Sun, Aug 5, 2018 at 1:11 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Alexander Mills wrote:
>
>> Yeah in this case what appeared to be happening is that if `git status` was
>> called concurrently, frequently I wouldnt get any stdout..
>
> Thanks for reporting.  A few questions:
>
> What platform are you on?  What version of Git are you running?  What
> is the output of "git version --build-options"?
>
> What is the exit status from these "git status" invocations that
> didn't write output?  What options are you passing to "git status"?
> Are there any other interesting symptoms?
>
> Is this part of a larger tool or script?  Are there other operations
> going on (e.g. something sending signals to these git processes)?
>
> Does the repository pass "git fsck"?  Any other symptoms or hints that
> could help in tracking this down?
>
> Can you paste a transcript of the commands or script you ran and what
> output it produced?  Is this something I should be able to reproduce?
>
> Thanks and hope that helps,
> Jonathan



-- 
Alexander D. Mills
¡¡¡ New cell phone number: (415)730-1805 !!!
alexander.d.mills@gmail.com

www.linkedin.com/pub/alexander-mills/b/7a5/418/

Re: concurrent access to multiple local git repos is error prone

[Alexander Mills]

To add something to the previous message, I have strong evidence that
the problem occurs when *different* repos are accessed concurrency,
not the same repo, as bizarre as that may be.

-alex

On Mon, Aug 6, 2018 at 12:36 AM, Alexander Mills
<alexander.d.mills@gmail.com> wrote:
> Hi Johnathan,
>
> Yeah this concurrency problem is real. Not only does it happen with
> `git status` the same thing happens with `git rev-parse
> --show-toplevel`.
> What happens is that I get no stdout when repos are accessed
> concurrently (and no stderr). If I limit concurrency to 1, the problem
> goes away. When I up the concurrency, the problem is sporadic, which
> is the exact signal for a concurrency/race-condition related issue.
> The signs are damn clear. I have seen this problem on MacOS I think a
> year back on a different project, but I never reported it b/c I hadn't
> really verified it.
>
> Like I said I am on Ubuntu. I have 3 git repos that are incorporated
> into the tool that's generating the problem. For one repo I got this:
>
> $ git fsck
>
> Checking object directories: 100% (256/256), done.
> dangling tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904
>
> For `$ git version --build-options` I have:
>
> git version 2.17.1
> cpu: x86_64
> no commit associated with this build
> sizeof-long: 8
>
>
> -alex
>
>
>
> On Sun, Aug 5, 2018 at 1:11 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>> Alexander Mills wrote:
>>
>>> Yeah in this case what appeared to be happening is that if `git status` was
>>> called concurrently, frequently I wouldnt get any stdout..
>>
>> Thanks for reporting.  A few questions:
>>
>> What platform are you on?  What version of Git are you running?  What
>> is the output of "git version --build-options"?
>>
>> What is the exit status from these "git status" invocations that
>> didn't write output?  What options are you passing to "git status"?
>> Are there any other interesting symptoms?
>>
>> Is this part of a larger tool or script?  Are there other operations
>> going on (e.g. something sending signals to these git processes)?
>>
>> Does the repository pass "git fsck"?  Any other symptoms or hints that
>> could help in tracking this down?
>>
>> Can you paste a transcript of the commands or script you ran and what
>> output it produced?  Is this something I should be able to reproduce?
>>
>> Thanks and hope that helps,
>> Jonathan
>
>
>
> --
> Alexander D. Mills
> ¡¡¡ New cell phone number: (415)730-1805 !!!
> alexander.d.mills@gmail.com
>
> www.linkedin.com/pub/alexander-mills/b/7a5/418/



-- 
Alexander D. Mills
¡¡¡ New cell phone number: (415)730-1805 !!!
alexander.d.mills@gmail.com

www.linkedin.com/pub/alexander-mills/b/7a5/418/

Re: concurrent access to multiple local git repos is error prone

[Jonathan Nieder]

(administrivia: please don't top-post)
Hi Alex,

Alexander Mills wrote:

> Yeah this concurrency problem is real. Not only does it happen with
> `git status` the same thing happens with `git rev-parse
> --show-toplevel`.

Sorry for the confusion --- I didn't mean to claim your experience was
not real!

What I wanted to make clear is that

 1. Git is designed to allow concurrent reads of a repository (and
    pushes to a repository).  If it doesn't work, that is a simple bug,
    not a design goal.

 2. Plenty of people rely on concurrently accessing repositories, so
    if it doesn't work, then (i) we definitely want to know and (ii)
    we're going to need a lot of detail to figure out what's happening,
    so we can fix it.

Does that make it clearer?

> What happens is that I get no stdout when repos are accessed
> concurrently (and no stderr). If I limit concurrency to 1, the problem
> goes away. When I up the concurrency, the problem is sporadic, which
> is the exact signal for a concurrency/race-condition related issue.
> The signs are damn clear. I have seen this problem on MacOS I think a
> year back on a different project, but I never reported it b/c I hadn't
> really verified it.
>
> Like I said I am on Ubuntu. I have 3 git repos that are incorporated
> into the tool that's generating the problem. For one repo I got this:
>
> $ git fsck
>
> Checking object directories: 100% (256/256), done.
> dangling tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904
>
> For `$ git version --build-options` I have:
>
> git version 2.17.1
> cpu: x86_64
> no commit associated with this build
> sizeof-long: 8

Thanks.  My best idea for a next step is that if you can come up with a
reproduction recipe, that would be very helpful.

It doesn't have to reproduce 100% of the time, but e.g. if you have a
script that reproduces it 50% of the time, I can run that script in a
loop.

Thanks,
Jonathan

Re: concurrent access to multiple local git repos is error prone

[Duy Nguyen]

On Mon, Aug 6, 2018 at 9:38 AM Alexander Mills
<alexander.d.mills@gmail.com> wrote:
> Yeah this concurrency problem is real. Not only does it happen with
> `git status` the same thing happens with `git rev-parse
> --show-toplevel`.

"git rev-parse --show-toplevel" having this same problem helps. This
command should never make any update in the repository, not even
taking any lock and very basic access to the repository (I think it
just needs to resolve HEAD, it does not even access object database).
It's so "simple" [1] that makes me think this problem is something not
really related to git. Perhaps the process is terminated abnormally
because it hits some system limits?

[1] well the repo discovery steps are a bit messy  but I think it's
unlikely we have any racing issues there.

> What happens is that I get no stdout when repos are accessed
> concurrently (and no stderr).
-- 
Duy