Re: Collective wisdom about repos on NFS accessed by concurrent clients (== corruption!?)

[Thomas Rast <trast@inf.ethz.ch>]

Kenneth Ölwing <kenneth@olwing.se> writes:

> On 2013-04-05 15:42, Thomas Rast wrote:
>> Can you run the same tests under strace or similar, and gather the
>> relevant outputs? Otherwise it's probably very hard to say what is
>> going wrong. In particular we've had some reports on lustre that
>> boiled down to "impossible" returns from libc functions, not git
>> issues. It's hard to say without some evidence. 
> Thomas, thanks for your reply.
>
> I'm assuming I should strace the git commands as they're issued? I'm
> already collecting regular stdout/err output in a log as I go. Is
> there any debugging things I can turn on to make the calls issue
> internal tracing of some sort?

I don't think there's any internal debugging that helps at this point.
Usually errors pointing to corruption are caused by a chain of syscalls
failing in some way, and the final error shows only the last one, so
strace() output is very interesting.

> The main issue I see is that I suspect it will generate so much data
> that it'll overflow my disk ;-).

Well, assuming you have some automated way of detecting when it fails,
you can just overwrite the same strace output file repeatedly; we're
only interested in the last one (or all the last ones if several gits
fail concurrently).

Fiddling with strace will unfortunately change the timings somewhat
(causing a bunch of extra context switches per syscall), but I hope that
you can still get it to reproduce.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch