[StGit PATCH] Read several objects at once with git cat-file --batch

[StGit PATCH] Read several objects at once with git cat-file --batch

[Karl Hasselström]

Instead of spawning a separate cat-file process for every blob and
commit we want to read. This speeds things up slightly: about 6-8%
when uncommitting and rebasing 1470 linux-kernel patches (perftest.py
rebase-newrebase-add-file-linux).

Signed-off-by: Karl Hasselström <kha@treskal.com>

---

 stgit/lib/git.py |   34 +++++++++++++++++++++++++++++++++-
 stgit/run.py     |   17 +++++++++++++++++
 2 files changed, 50 insertions(+), 1 deletions(-)


diff --git a/stgit/lib/git.py b/stgit/lib/git.py
index 648e190..95efd9a 100644
--- a/stgit/lib/git.py
+++ b/stgit/lib/git.py
@@ -520,6 +520,37 @@ class RunWithEnvCwd(RunWithEnv):
         @param args: Command and argument vector"""
         return RunWithEnv.run(self, args, self.env_in_cwd)
 
+class CatFileProcess(object):
+    def __init__(self, repo):
+        self.__repo = repo
+        self.__proc = None
+    def __get_process(self):
+        if not self.__proc:
+            self.__proc = self.__repo.run(['git', 'cat-file', '--batch']
+                                          ).run_background()
+        return self.__proc
+    def cat_file(self, sha1):
+        p = self.__get_process()
+        p.stdin.write('%s\n' % sha1)
+        p.stdin.flush()
+
+        # Read until we have the entire status line.
+        s = ''
+        while not '\n' in s:
+            s += os.read(p.stdout.fileno(), 4096)
+        h, b = s.split('\n', 1)
+        if h == '%s missing' % sha1:
+            raise SomeException()
+        hash, type, length = h.split()
+        assert hash == sha1
+        length = int(length)
+
+        # Read until we have the entire object plus the trailing
+        # newline.
+        while len(b) < length + 1:
+            b += os.read(p.stdout.fileno(), 4096)
+        return type, b[:-1]
+
 class Repository(RunWithEnv):
     """Represents a git repository."""
     def __init__(self, directory):
@@ -531,6 +562,7 @@ class Repository(RunWithEnv):
         self.__default_index = None
         self.__default_worktree = None
         self.__default_iw = None
+        self.__catfile = CatFileProcess(self)
     env = property(lambda self: { 'GIT_DIR': self.__git_dir })
     @classmethod
     def default(cls):
@@ -580,7 +612,7 @@ class Repository(RunWithEnv):
     directory = property(lambda self: self.__git_dir)
     refs = property(lambda self: self.__refs)
     def cat_object(self, sha1):
-        return self.run(['git', 'cat-file', '-p', sha1]).raw_output()
+        return self.__catfile.cat_file(sha1)[1]
     def rev_parse(self, rev):
         try:
             return self.get_commit(self.run(
diff --git a/stgit/run.py b/stgit/run.py
index 7493ed3..ccca059 100644
--- a/stgit/run.py
+++ b/stgit/run.py
@@ -130,6 +130,19 @@ class Run:
             raise self.exc('%s failed: %s' % (self.__cmd[0], e))
         self.__log_end(self.exitcode)
         self.__check_exitcode()
+    def __run_background(self):
+        """Run in background."""
+        assert self.__indata == None
+        try:
+            p = subprocess.Popen(self.__cmd, env = self.__env, cwd = self.__cwd,
+                                 stdin = subprocess.PIPE,
+                                 stdout = subprocess.PIPE,
+                                 stderr = subprocess.PIPE)
+        except OSError, e:
+            raise self.exc('%s failed: %s' % (self.__cmd[0], e))
+        self.stdin = p.stdin
+        self.stdout = p.stdout
+        self.stderr = p.stderr
     def returns(self, retvals):
         self.__good_retvals = retvals
         return self
@@ -181,6 +194,10 @@ class Run:
     def run(self):
         """Just run, with no IO redirection."""
         self.__run_noio()
+    def run_background(self):
+        """Run as a background process."""
+        self.__run_background()
+        return self
     def xargs(self, xargs):
         """Just run, with no IO redirection. The extra arguments are
         appended to the command line a few at a time; the command is

[StGit] kha/{safe,experimental} updated

[Karl Hasselström]

Catalin has pulled the safe and stable branches, but I've accumulated
some new stuff in kha/safe.

The stack log stuff (and nothing else) is still in kha/experimental.
It's unchanged since my last status mail, except for the addition of
an optimization at the end (will post as a follow-up to this mail).


                                 -+-


The following changes since commit 36a06e0194e013552499677e431e528ecb2faee9:
  Karl Hasselström (1):
        Global performance logging

are available in the git repository at:

  git://repo.or.cz/stgit/kha.git safe

Daniel White (7):
      Fix Makefile to correctly pass prefix option
      Remove variables regarding section 7 man pages
      Fix default install location for manpages
      Add install-doc target to makefile
      Add install-html target to makefile
      Remove installation of documentation from setup.py
      Updated INSTALL with documentation of Makefile

Karl Hasselström (1):
      Add some tests of refreshing removed files

Samuel Tardieu (2):
      Do not insert an empty line before the diffstat info
      Do not mess-up with commit message formatting when sending email

 Documentation/Makefile     |   25 +++++------
 INSTALL                    |   15 ++++--
 Makefile                   |   12 ++++-
 setup.py                   |    2 +-
 stgit/commands/mail.py     |    4 +-
 t/t2702-refresh-rm.sh      |  101 ++++++++++++++++++++++++++++++++++++++++++++
 templates/mailattch.tmpl   |    1 -
 templates/patchexport.tmpl |    1 -
 templates/patchmail.tmpl   |    1 -
 9 files changed, 135 insertions(+), 27 deletions(-)
 create mode 100755 t/t2702-refresh-rm.sh


                                 -+-


The following changes since commit 42857cbe036ba5917eacc9dbb5644d395f638ed9:
  Samuel Tardieu (1):
        Do not mess-up with commit message formatting when sending email

are available in the git repository at:

  git://repo.or.cz/stgit/kha.git experimental

Karl Hasselström (17):
      Write to a stack log when stack is modified
      New command: stg reset
      Log conflicts separately
      Log conflicts separately for all commands
      Add a --hard flag to stg reset
      Don't write a log entry if there were no changes
      Move stack reset function to a shared location
      New command: stg undo
      New command: stg redo
      Log and undo external modifications
      Make "stg log" show stack log instead of patch log
      Convert "stg refresh" to the new infrastructure
      New refresh tests
      Remove --undo flags from stg commands and docs
      Refactor stgit.commands.edit
      Implement "stg refresh --edit" again
      Read several objects at once with git cat-file --batch

 Documentation/tutorial.txt   |    4 +-
 TODO                         |    2 -
 stgit/commands/branch.py     |   19 +-
 stgit/commands/clone.py      |    2 +-
 stgit/commands/coalesce.py   |    2 +-
 stgit/commands/common.py     |   18 ++-
 stgit/commands/diff.py       |    6 +-
 stgit/commands/edit.py       |   82 +------
 stgit/commands/export.py     |    2 +-
 stgit/commands/files.py      |    6 +-
 stgit/commands/float.py      |    2 +-
 stgit/commands/fold.py       |    2 +-
 stgit/commands/goto.py       |    3 +-
 stgit/commands/hide.py       |    2 +-
 stgit/commands/id.py         |    2 +-
 stgit/commands/imprt.py      |    4 +-
 stgit/commands/log.py        |  169 +++++----------
 stgit/commands/mail.py       |    8 +-
 stgit/commands/new.py        |    3 +-
 stgit/commands/patches.py    |    2 +-
 stgit/commands/pick.py       |    2 +-
 stgit/commands/pop.py        |    4 +-
 stgit/commands/pull.py       |    2 +-
 stgit/commands/push.py       |   31 +--
 stgit/commands/rebase.py     |    4 +-
 stgit/commands/redo.py       |   52 ++++
 stgit/commands/refresh.py    |  338 ++++++++++++++++++---------
 stgit/commands/rename.py     |    2 +-
 stgit/commands/repair.py     |   11 +-
 stgit/commands/reset.py      |   57 +++++
 stgit/commands/resolved.py   |    2 +-
 stgit/commands/show.py       |    2 +-
 stgit/commands/sink.py       |    2 +-
 stgit/commands/status.py     |    3 +-
 stgit/commands/sync.py       |   26 +--
 stgit/commands/undo.py       |   49 ++++
 stgit/commands/unhide.py     |    2 +-
 stgit/git.py                 |    4 -
 stgit/lib/edit.py            |   99 ++++++++
 stgit/lib/git.py             |  108 ++++++++-
 stgit/lib/log.py             |  524 ++++++++++++++++++++++++++++++++++++++++++
 stgit/lib/stack.py           |   25 ++
 stgit/lib/transaction.py     |  125 +++++++----
 stgit/main.py                |    8 +
 stgit/run.py                 |   17 ++
 stgit/stack.py               |   45 +----
 stgit/utils.py               |   18 +-
 t/t1200-push-modified.sh     |    2 +-
 t/t1201-pull-trailing.sh     |    2 +-
 t/t1202-push-undo.sh         |    8 +-
 t/t1400-patch-history.sh     |  103 --------
 t/t2300-refresh-subdir.sh    |   29 +++-
 t/t2701-refresh-p.sh         |    2 +-
 t/t3100-reset.sh             |  160 +++++++++++++
 t/t3101-reset-hard.sh        |   56 +++++
 t/t3102-undo.sh              |   86 +++++++
 t/t3103-undo-hard.sh         |   56 +++++
 t/t3104-redo.sh              |  122 ++++++++++
 t/t3105-undo-external-mod.sh |   68 ++++++
 t/t3300-edit.sh              |    4 +-
 60 files changed, 1986 insertions(+), 614 deletions(-)
 create mode 100644 stgit/commands/redo.py
 create mode 100644 stgit/commands/reset.py
 create mode 100644 stgit/commands/undo.py
 create mode 100644 stgit/lib/edit.py
 create mode 100644 stgit/lib/log.py
 delete mode 100755 t/t1400-patch-history.sh
 create mode 100755 t/t3100-reset.sh
 create mode 100755 t/t3101-reset-hard.sh
 create mode 100755 t/t3102-undo.sh
 create mode 100755 t/t3103-undo-hard.sh
 create mode 100755 t/t3104-redo.sh
 create mode 100755 t/t3105-undo-external-mod.sh

-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle

Re: [StGit PATCH] Read several objects at once with git cat-file --batch

[Catalin Marinas]

2008/8/8 Karl Hasselström <kha@treskal.com>:
> Instead of spawning a separate cat-file process for every blob and
> commit we want to read. This speeds things up slightly: about 6-8%
> when uncommitting and rebasing 1470 linux-kernel patches (perftest.py
> rebase-newrebase-add-file-linux).

Which version of Git got the --batch option to git-cat-file? It might
be possible that default Git in Debian (testing) or Ubuntu doesn't
have this option. Maybe we could still have the original behaviour as
a fallback.

Otherwise, the patch looks allright. It took me a bit of time to see
why we need the new run_background() function (but in my current Git,
1.5.3.4.206.g58ba4, there wasn't such an option; I had to upgrade).

Thanks.

-- 
Catalin

Re: [StGit PATCH] Read several objects at once with git cat-file --batch

[Karl Hasselström]

On 2008-08-10 23:25:08 +0100, Catalin Marinas wrote:

> 2008/8/8 Karl Hasselström <kha@treskal.com>:
>
> > Instead of spawning a separate cat-file process for every blob and
> > commit we want to read. This speeds things up slightly: about 6-8%
> > when uncommitting and rebasing 1470 linux-kernel patches
> > (perftest.py rebase-newrebase-add-file-linux).
>
> Which version of Git got the --batch option to git-cat-file? It
> might be possible that default Git in Debian (testing) or Ubuntu
> doesn't have this option. Maybe we could still have the original
> behaviour as a fallback.

Hmm, I never realized it was so new. It's not in any 1.5.5 release,
but it is in 1.5.6-rc0. So realistically, we'll have to require
version 1.5.6 for this. (The patch should add a paragraph about the
required git version to some doc file.)

A fallback is certainly conceivable. The only nontrivial thing about
it would be to detect when it's necessary. And to make sure both code
paths are tested ...

(I have a similar patch that uses diff-tree --stdin, but that needs a
git with the patches I posted some hours ago.)

> Otherwise, the patch looks allright. It took me a bit of time to see
> why we need the new run_background() function (but in my current
> Git, 1.5.3.4.206.g58ba4, there wasn't such an option; I had to
> upgrade).

Thanks. I'm not 100% pleased with the background running stuff yet. It
works, but could use some refactoring so that callers don't have to
know too much.

-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle

Re: [StGit] kha/{safe,experimental} updated

[Catalin Marinas]

2008/8/8 Karl Hasselström <kha@treskal.com>:
> Catalin has pulled the safe and stable branches, but I've accumulated
> some new stuff in kha/safe.

I merged it, thanks.

> The stack log stuff (and nothing else) is still in kha/experimental.
> It's unchanged since my last status mail, except for the addition of
> an optimization at the end (will post as a follow-up to this mail).

I'm on holiday until Monday and I haven't got the time to test the
performance but we can probably move the experimental stuff into
master and start preparing some release candidates. I'll repost my
patches with the new git id format as well and we need a 0.14.4 in the
meantime (hopefully over the next two weeks).

Thanks.

-- 
Catalin

Re: [StGit] kha/{safe,experimental} updated

[Karl Hasselström]

On 2008-08-13 22:54:57 +0100, Catalin Marinas wrote:

> I'm on holiday until Monday and I haven't got the time to test the
> performance but we can probably move the experimental stuff into
> master and start preparing some release candidates.

Excellent.

> I'll repost my patches with the new git id format as well

Excellent. I was about to merge them a while back, but I noticed you
hadn't addressed the latest round of comments then.

> and we need a 0.14.4 in the meantime (hopefully over the next two
> weeks).

Excellent.

-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle

Re: [StGit] kha/{safe,experimental} updated

[Catalin Marinas]

2008/8/17 Karl Hasselström <kha@treskal.com>:
> On 2008-08-13 22:54:57 +0100, Catalin Marinas wrote:
>> I'll repost my patches with the new git id format as well
>
> Excellent. I was about to merge them a while back, but I noticed you
> hadn't addressed the latest round of comments then.

I actually decided to commit them since the comments were minor like
fixing the commit log or using strip_prefix(). I implemented them and
hopefully haven't forgotten any.

-- 
Catalin