From: Sam Vilain <sam@vilain.net>
To: Mike Hommey <mh@glandium.org>
Cc: git@vger.kernel.org, aroben@apple.com
Subject: Re: [PATCH 0/9] Make git-svn fetch ~1.7x faster
Date: Wed, 24 Oct 2007 13:43:51 +1300 [thread overview]
Message-ID: <471E9547.2030100@vilain.net> (raw)
In-Reply-To: <20071023060812.GA30978@glandium.org>
Mike Hommey wrote:
> On Mon, Oct 22, 2007 at 10:46:28PM -0700, Adam Roben wrote:
>> This patch series makes git-svn fetch about 1.7x faster by reducing the number
>> of forks/execs that occur for each file retrieved from Subversion. To do so, a
>> few new options are added to git-cat-file and git-hash-object to allow
>> continuous input on stdin and continuous output on stdout, so that one instance
>> of each of these commands can be kept running for the duration of the fetch.
>
> You don't need to do this to avoid forks. Just use git-fast-import
> instead.
git-fast-import only covers the hash-object side of things, not cat-file.
git-fast-import does not currently suit 'gradual deployment' for
converters such as git-svn, because it;
- returns object IDs at the end, when you checkpoint.
This could be 'fixed' by allowing a marks log file instead of or in
addition to the current behaviour, though if the exporter is
continually waiting for the tokens rather than using marks, it will
slow it down.
- you can't use plumbing commands, such as rev-parse, cat-file, etc on
objects which have not been checkpointed yet.
- can't just stream a file of unknown length to it as you can to
hash-object
These are the design trade-offs of using fast-import. Using
fast-import, you are creating a 'transaction' area which uses user
sequences instead of (git)database-issued identifiers. And this
transaction is isolated from the other concurrent users of the object
database. However the interface does not have the full git CLI
available to it, so unlike a regular database transaction, you end up
having to care.
Rewriting the importer so as to correctly deal with these problems is
quite challenging, and for slow import sources such as Subversion, of
limited merit.
Sam.
prev parent reply other threads:[~2007-10-24 0:44 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-10-23 5:46 [PATCH 0/9] Make git-svn fetch ~1.7x faster Adam Roben
2007-10-23 5:46 ` [PATCH 1/9] Add tests for git cat-file Adam Roben
2007-10-23 5:46 ` [PATCH 2/9] git-cat-file: Small refactor of cmd_cat_file Adam Roben
2007-10-23 5:46 ` [PATCH 3/9] git-cat-file: Make option parsing a little more flexible Adam Roben
2007-10-23 5:46 ` [PATCH 4/9] git-cat-file: Add --stdin option Adam Roben
2007-10-23 5:46 ` [PATCH 5/9] git-cat-file: Add --separator option Adam Roben
2007-10-23 5:46 ` [PATCH 6/9] Add tests for git hash-object Adam Roben
2007-10-23 5:46 ` [PATCH 7/9] git-hash-object: Add --stdin-paths option Adam Roben
2007-10-23 5:46 ` [PATCH 8/9] Git.pm: Add command_bidi_pipe and command_close_bidi_pipe Adam Roben
2007-10-23 5:46 ` [PATCH 9/9] git-svn: Make fetch ~1.7x faster Adam Roben
2007-10-23 7:01 ` Johannes Sixt
2007-10-24 6:34 ` Eric Wong
2007-10-24 6:48 ` Adam Roben
2007-10-23 5:53 ` [PATCH 7/9] git-hash-object: Add --stdin-paths option Shawn O. Pearce
2007-10-23 5:57 ` Adam Roben
2007-10-23 6:10 ` Shawn O. Pearce
2007-10-24 6:11 ` Eric Wong
2007-10-23 6:59 ` [PATCH 6/9] Add tests for git hash-object Johannes Sixt
2007-10-24 3:43 ` [PATCH 5/9] git-cat-file: Add --separator option Brian Downing
2007-10-24 4:26 ` Adam Roben
2007-10-23 6:59 ` [PATCH 1/9] Add tests for git cat-file Johannes Sixt
2007-10-23 6:08 ` [PATCH 0/9] Make git-svn fetch ~1.7x faster Mike Hommey
2007-10-23 6:13 ` Adam Roben
2007-10-24 0:43 ` Sam Vilain [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=471E9547.2030100@vilain.net \
--to=sam@vilain.net \
--cc=aroben@apple.com \
--cc=git@vger.kernel.org \
--cc=mh@glandium.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).