Re: Anyone know why git ls-remote output might be corrupted?

[Jeff King <peff@peff.net>]

On Sun, Jun 04, 2023 at 02:25:57AM -0400, Jeff King wrote:

> It does look like glibc's stdio will throw away buffer contents that get
> EAGAIN. Doing:
> 
>   perl -MFcntl -e '
>     fcntl(STDOUT, F_GETFL, $flags);
>     $flags |= O_NONBLOCK;
>     fcntl(STDOUT, F_SETFL, $flags);
>     exec @ARGV;
>   ' git ls-remote . | (sleep 1; tee output) | sha256sum
> 
> does result in some missing writes and broken input that looks like
> what's going on in this thread (in this case, it's writing to my
> terminal, which isn't fast enough to keep up; but you could also pipe to
> something like "tee output | sha256sum" to see that the output changes
> with each run). And naturally you'll need a big enough output from
> ls-remote to fill the pipe buffer.

Sorry for some slight confusion above. I edited my example command to
show piping to "sha256sum", but didn't modify the paragraph below it
(originally I was not piping anywhere, just sending to the terminal).

Adding the "sleep 1" means that the command will always fail (ls-remote
easily writes all of its output and hits EAGAIN before the other side
reads anything). But it means that the output is deterministic (the
first PIPE_BUF bytes make it through, and nothing else does). Removing
the sleep makes the output non-deterministic, but much more interesting
(you get missing chunks in the interior of the output).

So perhaps:

  perl ... git ls-remote . | tee output | sha256sum

is the most interesting case, because you'll get one of several possible
broken outputs, which you can identify by the changing sha256 output
(and then see the actual breakage by peeking at the "output" file).

-Peff