[RFC] Clone repositories recursive with depth 1

[RFC] Clone repositories recursive with depth 1

[Lars Schneider]

Hi,

I have a clean build machine and I want to clone my source code to this machine while transferring only the minimal necessary amount of data. Therefore I use this command:

git clone --recursive --depth 1 --single-branch <url>

Apparently this does not clone the submodules with "--depth 1" (using Git 2.4.9). As a workaround I tried:

git clone --depth 1 --single-branch <url>
cd <repo-name>
git submodule update --init --recursive --depth 1

However, this does not work either as I get:
fatal: reference is not a tree: <correct sha1 of the submodule referenced by the main project>
Unable to checkout <correct sha1 of the submodule referenced by the main project> in submodule path <submodule path>

How would you clone the repo? Is the behavior above expected? If not, should the "--depth 1" flag be applied recursively to all submodules on a clone --recursive? Has a patch implementing this a chance to get in?

Thanks,
Lars

Re: [RFC] Clone repositories recursive with depth 1

[Stefan Beller]

On Wed, Nov 11, 2015 at 6:09 AM, Lars Schneider
<larsxschneider@gmail.com> wrote:
> Hi,
>
> I have a clean build machine and I want to clone my source code to this machine while transferring only the minimal necessary amount of data. Therefore I use this command:
>
> git clone --recursive --depth 1 --single-branch <url>

That *should* work, actually.
However looking at the code it does not.

citing from builtin/clone.c:

    static struct option builtin_clone_options[] = {
        ...
        OPT_BOOL(0, "recursive", &option_recursive,
           N_("initialize submodules in the clone")),
        OPT_BOOL(0, "recurse-submodules", &option_recursive,
          N_("initialize submodules in the clone")),
        ...
    };
    ...
    static const char *argv_submodule[] = {
        "submodule", "update", "--init", "--recursive", NULL
    };

    if (!err && option_recursive)
        err = run_command_v_opt(argv_submodule, RUN_GIT_CMD);

So the --depth argument is not passed on, although "git submodule update"
definitely supports --depth.

In an upcoming series (next version of origin/sb/submodule-parallel-update),
this will slightly change, such it will be even easier to add the
depth argument in
there as we construct the argument list in code instead of hard coding
argv_submodule.

This may require some discussion whether you expect --depth to be recursed.
(What if you only want a top level shallow thing?, What if you want to have only
submodules shallow? What is the user expectation here?)

>
> Apparently this does not clone the submodules with "--depth 1" (using Git 2.4.9). As a workaround I tried:
>
> git clone --depth 1 --single-branch <url>
> cd <repo-name>
> git submodule update --init --recursive --depth 1
>
> However, this does not work either as I get:
> fatal: reference is not a tree: <correct sha1 of the submodule referenced by the main project>
> Unable to checkout <correct sha1 of the submodule referenced by the main project> in submodule path <submodule path>

That seems like another bug to me.

I just tried to clone a project and populate with submodules later and
it works as expected without these error messages.
(I am running some kind of xxx.dirty development version, most likely
origin/sb/submodule-parallel-update,
I'll check some other versions, too)

Re: [RFC] Clone repositories recursive with depth 1

[Stefan Beller]

On Wed, Nov 11, 2015 at 11:19 AM, Stefan Beller <sbeller@google.com> wrote:
> On Wed, Nov 11, 2015 at 6:09 AM, Lars Schneider
> <larsxschneider@gmail.com> wrote:
>> Hi,
>>
>> I have a clean build machine and I want to clone my source code to this machine while transferring only the minimal necessary amount of data. Therefore I use this command:
>>
>> git clone --recursive --depth 1 --single-branch <url>
>
> That *should* work, actually.
> However looking at the code it does not.
>
> citing from builtin/clone.c:
>
>     static struct option builtin_clone_options[] = {
>         ...
>         OPT_BOOL(0, "recursive", &option_recursive,
>            N_("initialize submodules in the clone")),
>         OPT_BOOL(0, "recurse-submodules", &option_recursive,
>           N_("initialize submodules in the clone")),
>         ...
>     };
>     ...
>     static const char *argv_submodule[] = {
>         "submodule", "update", "--init", "--recursive", NULL
>     };
>
>     if (!err && option_recursive)
>         err = run_command_v_opt(argv_submodule, RUN_GIT_CMD);
>
> So the --depth argument is not passed on, although "git submodule update"
> definitely supports --depth.
>
> In an upcoming series (next version of origin/sb/submodule-parallel-update),
> this will slightly change, such it will be even easier to add the
> depth argument in
> there as we construct the argument list in code instead of hard coding
> argv_submodule.
>
> This may require some discussion whether you expect --depth to be recursed.
> (What if you only want a top level shallow thing?, What if you want to have only
> submodules shallow? What is the user expectation here?)
>
>>
>> Apparently this does not clone the submodules with "--depth 1" (using Git 2.4.9). As a workaround I tried:
>>
>> git clone --depth 1 --single-branch <url>
>> cd <repo-name>
>> git submodule update --init --recursive --depth 1
>>

The workaround works with the origin/master version for me.

Notice the other email thread, which suggests to include --remote into the
call to  git submodule update depending on a branch config option being
present in the .gitmodules file.

Re: [RFC] Clone repositories recursive with depth 1

[Lars Schneider]

On 11 Nov 2015, at 21:09, Stefan Beller <sbeller@google.com> wrote:

> On Wed, Nov 11, 2015 at 11:19 AM, Stefan Beller <sbeller@google.com> wrote:
>> On Wed, Nov 11, 2015 at 6:09 AM, Lars Schneider
>> <larsxschneider@gmail.com> wrote:
>>> Hi,
>>> 
>>> I have a clean build machine and I want to clone my source code to this machine while transferring only the minimal necessary amount of data. Therefore I use this command:
>>> 
>>> git clone --recursive --depth 1 --single-branch <url>
>> 
>> That *should* work, actually.
>> However looking at the code it does not.
>> 
>> citing from builtin/clone.c:
>> 
>>    static struct option builtin_clone_options[] = {
>>        ...
>>        OPT_BOOL(0, "recursive", &option_recursive,
>>           N_("initialize submodules in the clone")),
>>        OPT_BOOL(0, "recurse-submodules", &option_recursive,
>>          N_("initialize submodules in the clone")),
>>        ...
>>    };
>>    ...
>>    static const char *argv_submodule[] = {
>>        "submodule", "update", "--init", "--recursive", NULL
>>    };
>> 
>>    if (!err && option_recursive)
>>        err = run_command_v_opt(argv_submodule, RUN_GIT_CMD);
>> 
>> So the --depth argument is not passed on, although "git submodule update"
>> definitely supports --depth.
>> 
>> In an upcoming series (next version of origin/sb/submodule-parallel-update),
>> this will slightly change, such it will be even easier to add the
>> depth argument in
>> there as we construct the argument list in code instead of hard coding
>> argv_submodule.
>> 
>> This may require some discussion whether you expect --depth to be recursed.
>> (What if you only want a top level shallow thing?, What if you want to have only
>> submodules shallow? What is the user expectation here?)
>> 
>>> 
>>> Apparently this does not clone the submodules with "--depth 1" (using Git 2.4.9). As a workaround I tried:
>>> 
>>> git clone --depth 1 --single-branch <url>
>>> cd <repo-name>
>>> git submodule update --init --recursive --depth 1
>>> 
> 
> The workaround works with the origin/master version for me.
> 
> Notice the other email thread, which suggests to include --remote into the
> call to  git submodule update depending on a branch config option being
> present in the .gitmodules file.

Can you check "[PATCH v2] add test to demonstrate that shallow recursive clones fail"? This demonstrates the failure that I see. I also tried the "--remote" flag but this does not work either (see test case).
Can you confirm this behavior?

Cheers,
Lars

Re: [RFC] Clone repositories recursive with depth 1

[Stefan Beller]

+cc Stanislav, who came up with the other thread for passing --remote
to git submodule

On Thu, Nov 12, 2015 at 1:39 AM, Lars Schneider
<larsxschneider@gmail.com> wrote:
>> Notice the other email thread, which suggests to include --remote into the
>> call to  git submodule update depending on a branch config option being
>> present in the .gitmodules file.
>
> Can you check "[PATCH v2] add test to demonstrate that shallow recursive clones fail"? This demonstrates the failure that I see. I also tried the "--remote" flag but this does not work either (see test case).
> Can you confirm this behavior?
>
> Cheers,
> Lars

I can confirm it breaks as expected here.

I may have confused you here by pointing to the --remote option.

(git clone is a bit stupid when it comes to submodule handling.)
All it does currently is this:

    if --recurseSubmodules option or --recursive option is given:
        run: "git submodule update --init --recursive"

No attention is paid to any other option such as --depth.
That's all I wanted to point out there.

Ideally we want to add:

    If there is a branch configured in the .gitmodules file,
    we would want to add the --remote command

    if we have given other options such as --depth or --reference
    we want to pass that along to the called submodule helper.

So I was looking at the internal code structure and think one of the next
series I am going to send will touch the code such that we can incorporate
the conditions as outlined above easier, because it is not hardcoded into an
array ["git", "submodule", "update" "--init", "--recursive"], as I
want to add yet
another dynamic option to the submodule helper invocation. (I want to add
--jobs <n> there)

Cheers,
Stefan

Re: [RFC] Clone repositories recursive with depth 1

[Fredrik Gustafsson]

On Wed, Nov 11, 2015 at 03:09:18PM +0100, Lars Schneider wrote:
> Apparently this does not clone the submodules with "--depth 1" (using Git 2.4.9). As a workaround I tried:
> 
> git clone --depth 1 --single-branch <url>
> cd <repo-name>
> git submodule update --init --recursive --depth 1
> 
> However, this does not work either as I get:
> fatal: reference is not a tree: <correct sha1 of the submodule referenced by the main project>
> Unable to checkout <correct sha1 of the submodule referenced by the main project> in submodule path <submodule path>

This looks like a familiar bug to me. I'm not sure if it's a bug or a
known behaviour. When the depth argument was introduced to submodules a
year (or two) ago there was a know bug. I not sure if it's fixed or
not.

The problem is that git is/was unable to fetch a sha1 but only a branch
or a tag. So fetching a submodule will fetch the HEAD of the submodule
with the requested depth. Then git will try to checkout a sha1 of that
submodule, which may or maynot exists.

Say that you fetch master of a submodule of depth 1 and do a checkout of
a commit that exists. When someonen else has pushed to that submodule,
the commit will not longer be reachable from depth 1 and if someone else
tries to clone with depth 1 it will fail with the same error message as
you got.

The solution to this is to allow git to fetch the sha1 the superproject
points to direct when fetching the submodule.

-- 
Fredrik Gustafsson

phone: +46 733-608274
e-mail: iveqy@iveqy.com
website: http://www.iveqy.com