git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Stefan Beller <sbeller@google.com>
To: git@vger.kernel.org
Cc: Stefan Beller <sbeller@google.com>
Subject: [RFC PATCHv1 00/12] git submodule update in C with parallel cloning
Date: Thu, 15 Oct 2015 18:52:01 -0700	[thread overview]
Message-ID: <1444960333-16003-1-git-send-email-sbeller@google.com> (raw)

So eventually we want to have projects with lots of submodules such as Android
(which would have O(1000) submodules).

The very first thing a user does is cloning a project, so we want to impress
with speed there as well. git clone however is lazy and just calls
`git submodule update --init --recursive`. So we need to make that fast.

This series rewrites parts of git submodule update in C and in the second-last
patch it separates cloning and doing the other actions(checkout/rebase/merge etc)
by doing the cloning first and then the rest.

The last patch (which is broken in the first version of the series), then 
proceeds to put the cloning of the submodules into the get_next_task callback
of the parallel process API.

That said, the first few patches introduce some churn in the behavior and tests
of Git, so maybe put your eyes there?

Thanks for any advice,
Stefan

Stefan Beller (12):
  git submodule update: Announce skipping submodules on stderr
  git submodule update: Announce uninitialized modules on stderr
  git submodule update: Move branch calculation to where it's needed
  git submodule update: Announce outcome of submodule operation to
    stderr
  git submodule update: Use its own list implementation.
  git submodule update: Handle unmerged submodules in C
  submodule config: keep update strategy around
  git submodule update: check for "none" in C
  git submodule update: Check url in C
  git submodule update: Clone projects from within C
  submodule--helper: Do not emit submodules to process directly.
  WIP/broken Clone all outstanding submodules in parallel

 builtin/submodule--helper.c | 221 ++++++++++++++++++++++++++++++++++++++++++++
 git-submodule.sh            |  38 +++-----
 submodule-config.c          |  11 +++
 submodule-config.h          |   1 +
 t/t7400-submodule-basic.sh  |  12 +--
 t/t7406-submodule-update.sh |  12 +--
 6 files changed, 256 insertions(+), 39 deletions(-)

-- 
2.5.0.277.gfdc362b.dirty

             reply	other threads:[~2015-10-16  1:52 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-10-16  1:52 Stefan Beller [this message]
2015-10-16  1:52 ` [PATCH 01/12] git submodule update: Announce skipping submodules on stderr Stefan Beller
2015-10-16 20:37   ` Junio C Hamano
2015-10-16 20:47     ` Stefan Beller
2015-10-16  1:52 ` [PATCH 02/12] git submodule update: Announce uninitialized modules " Stefan Beller
2015-10-16 20:54   ` Junio C Hamano
2015-10-16  1:52 ` [PATCH 03/12] git submodule update: Move branch calculation to where it's needed Stefan Beller
2015-10-16 20:54   ` Junio C Hamano
2015-10-16  1:52 ` [PATCH 04/12] git submodule update: Announce outcome of submodule operation to stderr Stefan Beller
2015-10-16  1:52 ` [PATCH 05/12] git submodule update: Use its own list implementation Stefan Beller
2015-10-16 21:02   ` Junio C Hamano
2015-10-16 21:08     ` Stefan Beller
2015-10-16  1:52 ` [PATCH 06/12] git submodule update: Handle unmerged submodules in C Stefan Beller
2015-10-20 21:11   ` Junio C Hamano
2015-10-20 21:21     ` Stefan Beller
2015-10-16  1:52 ` [PATCH 07/12] submodule config: keep update strategy around Stefan Beller
2015-10-16  1:52 ` [PATCH 08/12] git submodule update: check for "none" in C Stefan Beller
2015-10-16  1:52 ` [PATCH 09/12] git submodule update: Check url " Stefan Beller
2015-10-16  1:52 ` [PATCH 10/12] git submodule update: Clone projects from within C Stefan Beller
2015-10-16  1:52 ` [PATCH 11/12] submodule--helper: Do not emit submodules to process directly Stefan Beller
2015-10-16  1:52 ` [PATCH 12/12] WIP/broken Clone all outstanding submodules in parallel Stefan Beller

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1444960333-16003-1-git-send-email-sbeller@google.com \
    --to=sbeller@google.com \
    --cc=git@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).