Re: Collective wisdom about repos on NFS accessed by concurrent clients (== corruption!?)

["Kenneth Ölwing" <kenneth@olwing.se>]

On 2013-04-05 15:42, Thomas Rast wrote:
> Can you run the same tests under strace or similar, and gather the 
> relevant outputs? Otherwise it's probably very hard to say what is 
> going wrong. In particular we've had some reports on lustre that 
> boiled down to "impossible" returns from libc functions, not git 
> issues. It's hard to say without some evidence. 
Thomas, thanks for your reply.

I'm assuming I should strace the git commands as they're issued? I'm 
already collecting regular stdout/err output in a log as I go. Is there 
any debugging things I can turn on to make the calls issue internal 
tracing of some sort?

The main issue I see is that I suspect it will generate so much data 
that it'll overflow my disk ;-). Consider that my hammer consists of a 
Perl script that forks a number of tasks (e.g. 15) that each loops doing 
clone/commit/push/pull, with retrying on a few levels as errors occur 
(usually expected ones due to the concurrency, i.e. someone else pushed 
so a pull is necessary first, but occasionally the central repo is 
broken enough that it can't be cloned from, or at least not checked out 
master from...sometimes with printed errors that still give me a zero 
exit code...). That is then also run on several machines to the same 
repo to hopefully cause a breakage by sheer pounding...it's going to 
generate huge collections of strace output I expect...

I have some variations of this (e.g. all tasks are working on different 
branches, improving concurrency in some respects, but effects there have 
been that at the end I was missing a branch or so...). The likelihood of 
problems seems to increase when I actually use ssh in my ultimate setup, 
so a loadbalancer roundrobins each call to any of several hosts. In that 
case I must admit I don't know how to get in on the action since I guess 
I would need to strace the git-upload/receive-pack processes on the 
server side...?

Lastly, I don't know how much this will impact timings etc, or load. To 
get a broken result I have sometimes needed to run for many hours, 
others fairly quickly.

Well...I will try, it'll probably be a blast :-)

BTW, this is mostly done on Centos 6.3 and 6.4, locally built git-1.8.2.

ken1