[PATCH 0/2] clone: shallow-submodules should be single-branch by default

[PATCH 0/2] clone: shallow-submodules should be single-branch by default

[Bruce Perry via GitGitGadget]

I noticed a couple places where the behavior of recursive clones for shallow
submodules does not match what is implied by the documentation. Shallow
submodules should be, but aren't, single branch by default. It would also be
useful to allow users to override the shallow specification in gitmodules on
the command line for clones as they can for submodule update. The
modification here for the former is a bit ugly, but hopefully at least gets
the point across to start a discussion. First time submitting a patch here,
hopefully I'm getting the process right.

Bruce Perry (2):
  clone: shallow-submodules should be single-branch by default
  clone: no-shallow-submodules clone overrides option in gitmodules

 Documentation/git-clone.txt         |  3 ++
 Documentation/gitmodules.txt        |  4 +--
 builtin/clone.c                     | 10 ++++--
 t/t5614-clone-submodules-shallow.sh | 52 +++++++++++++++++++++++------
 4 files changed, 53 insertions(+), 16 deletions(-)


base-commit: daed0c68e94967bfbb3f87e15f7c9090dc1aa1e1
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1740%2Fbaperry2%2Fsubmods-clone-bug-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1740/baperry2/submods-clone-bug-v1
Pull-Request: https://github.com/git/git/pull/1740
-- 
gitgitgadget

[PATCH 1/2] clone: shallow-submodules should be single-branch by default

[Bruce Perry via GitGitGadget]

From: Bruce Perry <bruce.perry@nrel.gov>

Recursive cloning with with the `--shallow-submodules` command
line option (or the `shallow = true` option in .gitmodules) and
no specification of `--[no-]single-branch` does result in a
shallow submodule clone, but obtains all available branches.
This behavior is unexpected because in other circumstances
shallow clones imply single-branch clones. It also violates the
documented behavior: the documentation states that performing a
clone with `--recurse-submodules` is equivalent to doing a
nonrecursive clone then immediately running `git submodule
update --init --recursive`. However, with the `shallow = true`
option in .gitmodules, the former results in a shallow but not
single-branch clone of the submodules, while the latter results
submodules that are both shallow and single-branch.

Modify the logic for git clone with `--recurse-submodules` so
that if no option is specified for `--[no-]single-branch`, then
no `--[no-]single-branch` option is passed to the internal call
to `git submodule update --init --recursive`. As a result, the
default will be used, which is to make shallow clones also
single-branch and non-shallow clones multi-branch.

Modify the tests for shallow-submodules so that the submodule
has multiple branches and the expected behavior in terms of
obtaining branches is validated: shallow submodules must also
be single-branch, while non-shallow submodules must obtain
both branches.

Make a slight clarification to the documentation regarding
the above behavior, although primarily the effect is to bring
the behavior in line with the present documentation.

Signed-off-by: Bruce Perry <bruce.perry@nrel.gov>
---
 Documentation/git-clone.txt         |  3 +++
 Documentation/gitmodules.txt        |  4 ++--
 builtin/clone.c                     |  6 +++--
 t/t5614-clone-submodules-shallow.sh | 36 ++++++++++++++++++++---------
 4 files changed, 34 insertions(+), 15 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 5de18de2ab8..69248421390 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -297,6 +297,9 @@ or `--mirror` is given)
 
 `--`[`no-`]`shallow-submodules`::
 	All submodules which are cloned will be shallow with a depth of 1.
+	Also implies `--single-branch` for the submodule clones
+	unless `--no-single-branch` is passed, in which case histories near
+	the tips of all branches of the cloned submodules are fetched.
 
 `--`[`no-`]`remote-submodules`::
 	All submodules which are cloned will use the status of the submodule's
diff --git a/Documentation/gitmodules.txt b/Documentation/gitmodules.txt
index d9bec8b1875..c8160cd1df7 100644
--- a/Documentation/gitmodules.txt
+++ b/Documentation/gitmodules.txt
@@ -95,8 +95,8 @@ affected by this setting.
 
 submodule.<name>.shallow::
 	When set to true, a clone of this submodule will be performed as a
-	shallow clone (with a history depth of 1) unless the user explicitly
-	asks for a non-shallow clone.
+	shallow clone (single branch with a history depth of 1) unless the
+	user explicitly asks for a non-shallow clone.
 
 NOTES
 -----
diff --git a/builtin/clone.c b/builtin/clone.c
index b28f88eb43d..50ccce8902d 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -58,6 +58,7 @@ static const char * const builtin_clone_usage[] = {
 };
 
 static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
+static int option_single_branch_submodules;
 static int option_local = -1, option_no_hardlinks, option_shared;
 static int option_no_tags;
 static int option_shallow_submodules;
@@ -816,8 +817,8 @@ static int checkout(int submodule_progress, int filter_submodules)
 			strvec_pushf(&cmd.args, "--filter=%s",
 				     expand_list_objects_filter_spec(&filter_options));
 
-		if (option_single_branch >= 0)
-			strvec_push(&cmd.args, option_single_branch ?
+		if (option_single_branch_submodules >= 0)
+			strvec_push(&cmd.args, option_single_branch_submodules ?
 					       "--single-branch" :
 					       "--no-single-branch");
 
@@ -997,6 +998,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 
 	if (option_depth || option_since || option_not.nr)
 		deepen = 1;
+	option_single_branch_submodules = option_single_branch;
 	if (option_single_branch == -1)
 		option_single_branch = deepen ? 1 : 0;
 
diff --git a/t/t5614-clone-submodules-shallow.sh b/t/t5614-clone-submodules-shallow.sh
index c2a2bb453ee..b23c7d085aa 100755
--- a/t/t5614-clone-submodules-shallow.sh
+++ b/t/t5614-clone-submodules-shallow.sh
@@ -14,10 +14,14 @@ test_expect_success 'setup' '
 	mkdir sub &&
 	(
 		cd sub &&
-		git init &&
+		git init -b main &&
 		test_commit subcommit1 &&
 		test_commit subcommit2 &&
-		test_commit subcommit3
+		test_commit subcommit3 &&
+		git checkout -b branch &&
+		test_commit branchcommit1 &&
+		test_commit branchcommit2 &&
+		git checkout main
 	) &&
 	git submodule add "file://$pwd/sub" sub &&
 	git commit -m "add submodule"
@@ -30,16 +34,18 @@ test_expect_success 'nonshallow clone implies nonshallow submodule' '
 	git -C super_clone log --oneline >lines &&
 	test_line_count = 3 lines &&
 	git -C super_clone/sub log --oneline >lines &&
-	test_line_count = 3 lines
+	test_line_count = 3 lines &&
+	git -C super_clone/sub log --oneline --all >lines &&
+	test_line_count = 5 lines
 '
 
 test_expect_success 'shallow clone with shallow submodule' '
 	test_when_finished "rm -rf super_clone" &&
 	test_config_global protocol.file.allow always &&
 	git clone --recurse-submodules --depth 2 --shallow-submodules "file://$pwd/." super_clone &&
-	git -C super_clone log --oneline >lines &&
+	git -C super_clone log --oneline --all >lines &&
 	test_line_count = 2 lines &&
-	git -C super_clone/sub log --oneline >lines &&
+	git -C super_clone/sub log --oneline --all >lines &&
 	test_line_count = 1 lines
 '
 
@@ -50,7 +56,9 @@ test_expect_success 'shallow clone does not imply shallow submodule' '
 	git -C super_clone log --oneline >lines &&
 	test_line_count = 2 lines &&
 	git -C super_clone/sub log --oneline >lines &&
-	test_line_count = 3 lines
+	test_line_count = 3 lines  &&
+	git -C super_clone/sub log --oneline --all >lines &&
+	test_line_count = 5 lines
 '
 
 test_expect_success 'shallow clone with non shallow submodule' '
@@ -60,7 +68,9 @@ test_expect_success 'shallow clone with non shallow submodule' '
 	git -C super_clone log --oneline >lines &&
 	test_line_count = 2 lines &&
 	git -C super_clone/sub log --oneline >lines &&
-	test_line_count = 3 lines
+	test_line_count = 3 lines  &&
+	git -C super_clone/sub log --oneline --all >lines &&
+	test_line_count = 5 lines
 '
 
 test_expect_success 'non shallow clone with shallow submodule' '
@@ -69,7 +79,7 @@ test_expect_success 'non shallow clone with shallow submodule' '
 	git clone --recurse-submodules --no-local --shallow-submodules "file://$pwd/." super_clone &&
 	git -C super_clone log --oneline >lines &&
 	test_line_count = 3 lines &&
-	git -C super_clone/sub log --oneline >lines &&
+	git -C super_clone/sub log --oneline --all >lines &&
 	test_line_count = 1 lines
 '
 
@@ -87,7 +97,7 @@ test_expect_success 'clone follows shallow recommendation' '
 	) &&
 	(
 		cd super_clone/sub &&
-		git log --oneline >lines &&
+		git log --oneline --all >lines &&
 		test_line_count = 1 lines
 	)
 '
@@ -105,7 +115,9 @@ test_expect_success 'get unshallow recommended shallow submodule' '
 	(
 		cd super_clone/sub &&
 		git log --oneline >lines &&
-		test_line_count = 3 lines
+		test_line_count = 3 lines &&
+		git log --oneline --all >lines &&
+		test_line_count = 5 lines
 	)
 '
 
@@ -124,7 +136,9 @@ test_expect_success 'clone follows non shallow recommendation' '
 	(
 		cd super_clone/sub &&
 		git log --oneline >lines &&
-		test_line_count = 3 lines
+		test_line_count = 3 lines &&
+		git log --oneline --all >lines &&
+		test_line_count = 5 lines
 	)
 '
 
-- 
gitgitgadget

[PATCH 2/2] clone: no-shallow-submodules clone overrides option in gitmodules

[Bruce Perry via GitGitGadget]

From: Bruce Perry <bruce.perry@nrel.gov>

When the `--no-shallow-submodules` option is used for cloning, pass
the `--no-recommend-shallow` option to the internal submodule update
command, which will override an option for shallow in .gitmodules
if present. This allows the user to force a non-shallow clone of
the submodules if desired. It brings the behavior in line with
the present documentation, which states the option in gitmodules is
used unless explitcitly overridden by the user. Add a test to confirm
that the override is done when using the clone command (similar to
a current test that validates similar behavior for the submodule
update command).

Signed-off-by: Bruce Perry <bruce.perry@nrel.gov>
---
 builtin/clone.c                     |  4 +++-
 t/t5614-clone-submodules-shallow.sh | 16 ++++++++++++++++
 2 files changed, 19 insertions(+), 1 deletion(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index 50ccce8902d..63cfb48484c 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -61,7 +61,7 @@ static int option_no_checkout, option_bare, option_mirror, option_single_branch
 static int option_single_branch_submodules;
 static int option_local = -1, option_no_hardlinks, option_shared;
 static int option_no_tags;
-static int option_shallow_submodules;
+static int option_shallow_submodules = -1;
 static int option_reject_shallow = -1;    /* unspecified */
 static int config_reject_shallow = -1;    /* unspecified */
 static int deepen;
@@ -798,6 +798,8 @@ static int checkout(int submodule_progress, int filter_submodules)
 
 		if (option_shallow_submodules == 1)
 			strvec_push(&cmd.args, "--depth=1");
+		else if (option_shallow_submodules == 0)
+			strvec_push(&cmd.args, "--no-recommend-shallow");
 
 		if (max_jobs != -1)
 			strvec_pushf(&cmd.args, "--jobs=%d", max_jobs);
diff --git a/t/t5614-clone-submodules-shallow.sh b/t/t5614-clone-submodules-shallow.sh
index b23c7d085aa..8e2965248a2 100755
--- a/t/t5614-clone-submodules-shallow.sh
+++ b/t/t5614-clone-submodules-shallow.sh
@@ -102,6 +102,22 @@ test_expect_success 'clone follows shallow recommendation' '
 	)
 '
 
+test_expect_success 'no-shallow-submodules clone option overrides gitmodules' '
+	test_when_finished "rm -rf super_clone" &&
+	test_config_global protocol.file.allow always &&
+	git clone --recurse-submodules --no-shallow-submodules --no-local "file://$pwd/." super_clone &&
+	(
+		cd super_clone &&
+		git log --oneline >lines &&
+		test_line_count = 4 lines
+	) &&
+	(
+		cd super_clone/sub &&
+		git log --oneline --all >lines &&
+		test_line_count = 5 lines
+	)
+'
+
 test_expect_success 'get unshallow recommended shallow submodule' '
 	test_when_finished "rm -rf super_clone" &&
 	test_config_global protocol.file.allow always &&
-- 
gitgitgadget

RE: [PATCH 0/2] clone: shallow-submodules should be single-branch by default

[rsbecker]

On Tuesday, July 2, 2024 3:08 PM, Bruce Perry wrote:
>I noticed a couple places where the behavior of recursive clones for shallow
>submodules does not match what is implied by the documentation. Shallow
>submodules should be, but aren't, single branch by default. It would also be useful
>to allow users to override the shallow specification in gitmodules on the command
>line for clones as they can for submodule update. The modification here for the
>former is a bit ugly, but hopefully at least gets the point across to start a discussion.
>First time submitting a patch here, hopefully I'm getting the process right.
>
>Bruce Perry (2):
>  clone: shallow-submodules should be single-branch by default
>  clone: no-shallow-submodules clone overrides option in gitmodules
>
> Documentation/git-clone.txt         |  3 ++
> Documentation/gitmodules.txt        |  4 +--
> builtin/clone.c                     | 10 ++++--
> t/t5614-clone-submodules-shallow.sh | 52 +++++++++++++++++++++++------
> 4 files changed, 53 insertions(+), 16 deletions(-)
>
>
>base-commit: daed0c68e94967bfbb3f87e15f7c9090dc1aa1e1
>Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-
>1740%2Fbaperry2%2Fsubmods-clone-bug-v1
>Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-
>1740/baperry2/submods-clone-bug-v1
>Pull-Request: https://github.com/git/git/pull/1740

I am concerned about this one. Many CI systems (including Jenkins GitSCM) assume a detached head for submodule clone/checkout. Adding a branch to the mix will change the expected semantics. Am I missing something?
--Randall

Re: [PATCH 0/2] clone: shallow-submodules should be single-branch by default

[Bruce Perry]

Randall,

Perhaps I was using imprecise terminology. This change should not
impact whether submodule clones land in a detached head state, so it
should not impact anything that assumes submodule clones are detached
head.

The change being made is this: "git clone --recurse-submodules
--shallow-submodules" currently gives you a submodule with a detached
head at the desired state, but also downloads data for the tips of all
branches in the remote being cloned (potentially a lot of unneeded
data as in my use case). The modification means the same command would
give you a detached head at the desired state, plus the tip of only
the default branch in the remote. The modified behavior matches the
current behavior for a simple "git clone" followed by "git submodule
update --init --recurse--submodules --depth=1".

Thanks,
Bruce

(Resent due to a formatting failure)

On Tue, Jul 2, 2024, 1:12 PM <rsbecker@nexbridge.com> wrote:
>
> On Tuesday, July 2, 2024 3:08 PM, Bruce Perry wrote:
> >I noticed a couple places where the behavior of recursive clones for shallow
> >submodules does not match what is implied by the documentation. Shallow
> >submodules should be, but aren't, single branch by default. It would also be useful
> >to allow users to override the shallow specification in gitmodules on the command
> >line for clones as they can for submodule update. The modification here for the
> >former is a bit ugly, but hopefully at least gets the point across to start a discussion.
> >First time submitting a patch here, hopefully I'm getting the process right.
> >
> >Bruce Perry (2):
> >  clone: shallow-submodules should be single-branch by default
> >  clone: no-shallow-submodules clone overrides option in gitmodules
> >
> > Documentation/git-clone.txt         |  3 ++
> > Documentation/gitmodules.txt        |  4 +--
> > builtin/clone.c                     | 10 ++++--
> > t/t5614-clone-submodules-shallow.sh | 52 +++++++++++++++++++++++------
> > 4 files changed, 53 insertions(+), 16 deletions(-)
> >
> >
> >base-commit: daed0c68e94967bfbb3f87e15f7c9090dc1aa1e1
> >Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-
> >1740%2Fbaperry2%2Fsubmods-clone-bug-v1
> >Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-
> >1740/baperry2/submods-clone-bug-v1
> >Pull-Request: https://github.com/git/git/pull/1740
>
> I am concerned about this one. Many CI systems (including Jenkins GitSCM) assume a detached head for submodule clone/checkout. Adding a branch to the mix will change the expected semantics. Am I missing something?
> --Randall
>

RE: [PATCH 0/2] clone: shallow-submodules should be single-branch by default

[rsbecker]

On Wednesday, July 3, 2024 1:00 AM, Bruce Perry wrote:
>Perhaps I was using imprecise terminology. This change should not impact whether
>submodule clones land in a detached head state, so it should not impact anything
>that assumes submodule clones are detached head.
>
>The change being made is this: "git clone --recurse-submodules --shallow-
>submodules" currently gives you a submodule with a detached head at the desired
>state, but also downloads data for the tips of all branches in the remote being
>cloned (potentially a lot of unneeded data as in my use case). The modification
>means the same command would give you a detached head at the desired state,
>plus the tip of only the default branch in the remote. The modified behavior
>matches the current behavior for a simple "git clone" followed by "git submodule
>update --init --recurse--submodules --depth=1".
>
>Thanks,
>Bruce
>
>(Resent due to a formatting failure)
>
>On Tue, Jul 2, 2024, 1:12 PM <rsbecker@nexbridge.com> wrote:
>>
>> On Tuesday, July 2, 2024 3:08 PM, Bruce Perry wrote:
>> >I noticed a couple places where the behavior of recursive clones for
>> >shallow submodules does not match what is implied by the
>> >documentation. Shallow submodules should be, but aren't, single
>> >branch by default. It would also be useful to allow users to override
>> >the shallow specification in gitmodules on the command line for
>> >clones as they can for submodule update. The modification here for the former is
>a bit ugly, but hopefully at least gets the point across to start a discussion.
>> >First time submitting a patch here, hopefully I'm getting the process right.
>> >
>> >Bruce Perry (2):
>> >  clone: shallow-submodules should be single-branch by default
>> >  clone: no-shallow-submodules clone overrides option in gitmodules
>> >
>> > Documentation/git-clone.txt         |  3 ++
>> > Documentation/gitmodules.txt        |  4 +--
>> > builtin/clone.c                     | 10 ++++--
>> > t/t5614-clone-submodules-shallow.sh | 52
>> > +++++++++++++++++++++++------
>> > 4 files changed, 53 insertions(+), 16 deletions(-)
>> >
>> >
>> >base-commit: daed0c68e94967bfbb3f87e15f7c9090dc1aa1e1
>> >Published-As:
>> >https://github.com/gitgitgadget/git/releases/tag/pr-git-
>> >1740%2Fbaperry2%2Fsubmods-clone-bug-v1
>> >Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-
>> >1740/baperry2/submods-clone-bug-v1
>> >Pull-Request: https://github.com/git/git/pull/1740
>>
>> I am concerned about this one. Many CI systems (including Jenkins GitSCM)
>assume a detached head for submodule clone/checkout. Adding a branch to the mix
>will change the expected semantics. Am I missing something?

Thanks for clearing that up. Fine with me.
Regards
Randall

Re: [PATCH 0/2] clone: shallow-submodules should be single-branch by default

[Bruce Perry]

Any further comments on this or changes I should make so it can be merged?

The current code behavior does not match the documentation, so
something should be changed. If these changes are not desired, I could
submit an alternative patch that changes the documentation to clarify
what currently happens instead.


On Wed, Jul 3, 2024 at 8:05 AM <rsbecker@nexbridge.com> wrote:
>
> On Wednesday, July 3, 2024 1:00 AM, Bruce Perry wrote:
> >Perhaps I was using imprecise terminology. This change should not impact whether
> >submodule clones land in a detached head state, so it should not impact anything
> >that assumes submodule clones are detached head.
> >
> >The change being made is this: "git clone --recurse-submodules --shallow-
> >submodules" currently gives you a submodule with a detached head at the desired
> >state, but also downloads data for the tips of all branches in the remote being
> >cloned (potentially a lot of unneeded data as in my use case). The modification
> >means the same command would give you a detached head at the desired state,
> >plus the tip of only the default branch in the remote. The modified behavior
> >matches the current behavior for a simple "git clone" followed by "git submodule
> >update --init --recurse--submodules --depth=1".
> >
> >Thanks,
> >Bruce
> >
> >(Resent due to a formatting failure)
> >
> >On Tue, Jul 2, 2024, 1:12 PM <rsbecker@nexbridge.com> wrote:
> >>
> >> On Tuesday, July 2, 2024 3:08 PM, Bruce Perry wrote:
> >> >I noticed a couple places where the behavior of recursive clones for
> >> >shallow submodules does not match what is implied by the
> >> >documentation. Shallow submodules should be, but aren't, single
> >> >branch by default. It would also be useful to allow users to override
> >> >the shallow specification in gitmodules on the command line for
> >> >clones as they can for submodule update. The modification here for the former is
> >a bit ugly, but hopefully at least gets the point across to start a discussion.
> >> >First time submitting a patch here, hopefully I'm getting the process right.
> >> >
> >> >Bruce Perry (2):
> >> >  clone: shallow-submodules should be single-branch by default
> >> >  clone: no-shallow-submodules clone overrides option in gitmodules
> >> >
> >> > Documentation/git-clone.txt         |  3 ++
> >> > Documentation/gitmodules.txt        |  4 +--
> >> > builtin/clone.c                     | 10 ++++--
> >> > t/t5614-clone-submodules-shallow.sh | 52
> >> > +++++++++++++++++++++++------
> >> > 4 files changed, 53 insertions(+), 16 deletions(-)
> >> >
> >> >
> >> >base-commit: daed0c68e94967bfbb3f87e15f7c9090dc1aa1e1
> >> >Published-As:
> >> >https://github.com/gitgitgadget/git/releases/tag/pr-git-
> >> >1740%2Fbaperry2%2Fsubmods-clone-bug-v1
> >> >Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-
> >> >1740/baperry2/submods-clone-bug-v1
> >> >Pull-Request: https://github.com/git/git/pull/1740
> >>
> >> I am concerned about this one. Many CI systems (including Jenkins GitSCM)
> >assume a detached head for submodule clone/checkout. Adding a branch to the mix
> >will change the expected semantics. Am I missing something?
>
> Thanks for clearing that up. Fine with me.
> Regards
> Randall
>