git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Jeff King <peff@peff.net>
To: steve.norman@thomsonreuters.com
Cc: Junio C Hamano <gitster@pobox.com>,
	pclouds@gmail.com, git@vger.kernel.org
Subject: Re: [PATCH] index-pack: avoid excessive re-reading of pack directory
Date: Tue, 9 Jun 2015 13:41:06 -0400	[thread overview]
Message-ID: <20150609174106.GB32034@peff.net> (raw)
In-Reply-To: <20150609172436.GA32034@peff.net>

On Tue, Jun 09, 2015 at 01:24:36PM -0400, Jeff King wrote:

> I tested this on my system, and confirmed that for a "git clone
> --no-local --bare git.git":
> 
>   1. It cuts the number of openat/getdents/close syscalls by several
>      orders of magnitude.
> 
>   2. The overall time drops from ~11.4s to ~10.5s. I suppose if I timed
>      only the `index-pack` process, it would be even higher (as a
>      percentage improvement).

Just for fun, I did a "git pack-objects --all --stdout" from linux.git,
and then timed "git index-pack --stdin" on it in an empty repo. With a
configured alternate pointing to another empty repo, just to make it
more unfair. And then I stored it all on a ramdisk to emphasize the cost
of the syscalls versus hitting the disk. The numbers I got were:

  [before]
  real    2m13.093s
  user    3m31.884s
  sys     0m55.208s

  [after]
  real    1m40.389s
  user    3m10.776s
  sys     0m26.012s

That's sort of a ridiculous test, but it does show that this was having
some impact even on "normal" systems without insane syscall latencies.

-Peff

  reply	other threads:[~2015-06-09 17:41 UTC|newest]

Thread overview: 32+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-05-21 13:13 Troubleshoot clone issue to NFS steve.norman
2015-05-21 14:00 ` Christian Couder
2015-05-21 14:31 ` Duy Nguyen
2015-05-21 14:38   ` Duy Nguyen
2015-05-21 15:53     ` steve.norman
2015-05-22  0:16       ` Duy Nguyen
2015-05-22  7:12         ` Jeff King
2015-05-22  8:35           ` steve.norman
2015-05-22 10:05             ` Duy Nguyen
2015-05-22 14:37               ` Junio C Hamano
2015-05-22 15:02               ` steve.norman
2015-05-22 23:51                 ` [PATCH/RFC 0/3] using stat() to avoid re-scanning pack dir Jeff King
2015-05-22 23:51                   ` [PATCH 1/3] stat_validity: handle non-regular files Jeff King
2015-05-23 11:00                     ` Michael Haggerty
2015-05-24  8:29                       ` Jeff King
2015-05-22 23:52                   ` [PATCH 2/3] cache.h: move stat_validity definition up Jeff King
2015-05-22 23:54                   ` [PATCH 3/3] prepare_packed_git: use stat_validity to avoid re-reading packs Jeff King
2015-05-23  1:19                   ` [PATCH/RFC 0/3] using stat() to avoid re-scanning pack dir Duy Nguyen
2015-05-23  1:21                     ` Duy Nguyen
2015-05-24  8:20                     ` Jeff King
2015-05-24  9:00           ` Troubleshoot clone issue to NFS Duy Nguyen
2015-06-05 12:01             ` steve.norman
2015-06-05 12:18               ` Jeff King
2015-06-05 12:29                 ` [PATCH] index-pack: avoid excessive re-reading of pack directory Jeff King
2015-06-09 17:24                   ` Jeff King
2015-06-09 17:41                     ` Jeff King [this message]
2015-06-10  3:46                   ` Shawn Pearce
2015-06-10 14:00                     ` Jeff King
2015-06-10 14:36                       ` Duy Nguyen
2015-06-10 21:34                       ` Shawn Pearce
2015-06-05 14:20                 ` Troubleshoot clone issue to NFS steve.norman
2015-06-16 20:50                 ` Jeff King

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20150609174106.GB32034@peff.net \
    --to=peff@peff.net \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=pclouds@gmail.com \
    --cc=steve.norman@thomsonreuters.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).