pull from stdin

pull from stdin

[miket99]

Hello,

I am using a rather complicated distributed workflow to keep various git repositories in sync. Basically I am creating bundles, pushing them to a server, and pull them again using a special application which output the bundle content to stdin. Unfortunately git-fetch and friends do not like /dev/stdin as input. Is there any way to pass the bundle contents via stdin to git?

Minimal sample:
~$ mkdir test
~$ cd test
~/test$ git init
Initialized empty Git repository in /home/test/test/.git/
~/test$ echo a > a
~/test$ git add -A
~/test$ git commit -m test
[master (root-commit) 4ed639a] test
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 a
~/test$ git bundle create b --all
Counting objects: 3, done.
Writing objects: 100% (3/3), 193 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
~/test$ cat b | git pull -- /dev/stdin
fatal: '/dev/stdin' does not appear to be a git repository
fatal: The remote end hung up unexpectedly

Re: pull from stdin

[Jeff King]

On Thu, Jul 07, 2011 at 11:23:04AM -0400, miket99@Safe-mail.net wrote:

> I am using a rather complicated distributed workflow to keep various
> git repositories in sync. Basically I am creating bundles, pushing
> them to a server, and pull them again using a special application
> which output the bundle content to stdin. Unfortunately git-fetch and
> friends do not like /dev/stdin as input. Is there any way to pass the
> bundle contents via stdin to git?
> [...]
> ~/test$ cat b | git pull -- /dev/stdin
> fatal: '/dev/stdin' does not appear to be a git repository
> fatal: The remote end hung up unexpectedly

I think the problem is that "git fetch" tests for a bundle by checking
whether the argument is a regular file, which /dev/stdin is not. We
could perhaps fix that, but I think it still wouldn't work. The
bundle-reading code calls lseek, and your pipe will not be seekable.

So this works:

  $ git bundle unbundle /dev/stdin <b
  1c0138bbb920cdbf8e80bea38c4f77b0d01c3763 refs/heads/master
  1c0138bbb920cdbf8e80bea38c4f77b0d01c3763 HEAD

but this doesn't:

  $ cat b | git bundle unbundle /dev/stdin
  fatal: early EOF
  error: index-pack died

I don't think the problem is in the bundle format. I think it is simply
an issue that we read the bundle header in one process via buffered I/O,
and then use seek to pass the rest on to index-pack in another process.
So you could solve it by either:

  1. Using unbuffered I/O and reading a character at a time from the
     bundle when the input is not seekable. This will involve many more
     syscalls, of course, but we would only have to do it for the list
     of refs, not for the actual pack data.

  2. Instead of passing the descriptor to index-pack, open a pipe to
     index-pack and pass the data through the original process. For
     the sake of efficiency, we could enable this only when the input
     isn't seekable.

-Peff