[PATCH] fetch-pack: test: demonstrate segmentation fault when run with fsckObjects but without --lock-pack

[PATCH] fetch-pack: test: demonstrate segmentation fault when run with fsckObjects but without --lock-pack

[Kirill Smelkov]

When running git-backup[1] over lab.nexedi.com archive we noticed that
Git started to crash on fetch-pack operation[2] after Git upgrade. We
tracked this down to be a regression introduced by Git 2.31, more
specifically by commit 5476e1efded5 (fetch-pack: print and use dangling
.gitmodules), which started to index and lock packfiles on do_keep=y &&
fsck_objects=y even if pack_lockfiles=NULL, which leads to
NULL-dereference when trying to append an entry to that pack_lockfiles
stringlist.

Please find attached a test that demonstrates this problem.

When run with test_expect_failure changed to test_expect_success the test
crashes with

    ./test-lib.sh: line 1063: 599675 Segmentation fault      (core dumped) git -c fetch.fsckObjects=true -c fetch.unpackLimit=1 -C client fetch-pack ../server $(git -C server rev-parse refs/heads/main)

and the backtrace is

    Program terminated with signal SIGSEGV, Segmentation fault.
    #0  0x0000558032c7ef3b in string_list_append_nodup (list=0x0,
        string=0x558033b695f0 ".git/objects/pack/pack-c1b2455a361bb50a0db087e7202db73d62938fa1.keep") at string-list.c:218
    218             ALLOC_GROW(list->items, list->nr + 1, list->alloc);
    (gdb) bt
    #0  0x0000558032c7ef3b in string_list_append_nodup (list=0x0,
        string=0x558033b695f0 ".git/objects/pack/pack-c1b2455a361bb50a0db087e7202db73d62938fa1.keep") at string-list.c:218
    #1  0x0000558032b5b83f in get_pack (args=0x7ffe680f3fa0, xd=0x7ffe680f4058, pack_lockfiles=0x0, index_pack_args=0x0,
        sought=0x558033b65e90, nr_sought=1, gitmodules_oids=0x558032e17b88 <fsck_options+72>) at fetch-pack.c:1042
    #2  0x0000558032b5e0b3 in do_fetch_pack_v2 (args=0x7ffe680f3fa0, fd=0x7ffe680f4058, orig_ref=0x558033b67b90, sought=0x558033b65e90,
        nr_sought=1, shallows=0x7ffe680f3df0, si=0x7ffe680f3e10, pack_lockfiles=0x0) at fetch-pack.c:1781
    #3  0x0000558032b5ef3c in fetch_pack (args=0x7ffe680f3fa0, fd=0x7ffe680f4058, ref=0x558033b67b90, sought=0x558033b65e90, nr_sought=1,
        shallow=0x7ffe680f3f80, pack_lockfiles=0x0, version=protocol_v2) at fetch-pack.c:2073
    #4  0x0000558032a08851 in cmd_fetch_pack (argc=3, argv=0x7ffe680f4500, prefix=0x0) at builtin/fetch-pack.c:242
    #5  0x00005580329b2be3 in run_builtin (p=0x558032e0bb30 <commands+1008>, argc=3, argv=0x7ffe680f4500) at git.c:474
    #6  0x00005580329b2ffe in handle_builtin (argc=3, argv=0x7ffe680f4500) at git.c:729
    #7  0x00005580329b3222 in run_argv (argcp=0x7ffe680f433c, argv=0x7ffe680f4330) at git.c:793
    #8  0x00005580329b3796 in cmd_main (argc=3, argv=0x7ffe680f4500) at git.c:928
    #9  0x0000558032ab9002 in main (argc=10, argv=0x7ffe680f44c8) at common-main.c:62

A simple debug patch below

    --- a/fetch-pack.c
    +++ b/fetch-pack.c
    @@ -1032,7 +1032,7 @@ static int get_pack(struct fetch_pack_args *args,
            cmd.git_cmd = 1;
            if (start_command(&cmd))
                    die(_("fetch-pack: unable to fork off %s"), cmd_name);
    -       if (do_keep && (pack_lockfiles || fsck_objects)) {
    +       if (do_keep && (pack_lockfiles /*|| fsck_objects*/)) {
                    int is_well_formed;
                    char *pack_lockfile = index_pack_lockfile(cmd.out, &is_well_formed);

makes the crash go away, but I did not investigated what should be the
proper logic.

Thanks beforehand for fixing this NULL-pointer dereference.

Kirill

[1] https://lab.nexedi.com/kirr/git-backup
[2] https://lab.nexedi.com/kirr/git-backup/-/blob/3230197c/git-backup.go#L717-739

Cc: Jonathan Tan <jonathantanmy@google.com>
Cc: Alain Takoudjou <alain.takoudjou@nexedi.com>
Cc: Jérome Perrin <jerome@nexedi.com>
Signed-off-by: Kirill Smelkov <kirr@nexedi.com>
---
 t/t5500-fetch-pack.sh | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
index 1bc15a3f08..e3dbe79613 100755
--- a/t/t5500-fetch-pack.sh
+++ b/t/t5500-fetch-pack.sh
@@ -1056,6 +1056,20 @@ test_expect_success 'fetch with --filter=blob:limit=0' '
 	fetch_filter_blob_limit_zero server server
 '

+test_expect_failure 'fetch with fsckObjects but without --lock-pack does not segfault' '
+	rm -rf server client &&
+	git init server &&
+	test_commit -C server 1 &&
+
+	git init client &&
+	# unpackLimit=1 forces to keep received pack as pack instead of
+	# unpacking it to loose objects. The code is currently segfaulting when
+	# trying to index that kept pack.
+	git -c fetch.fsckObjects=true -c fetch.unpackLimit=1 \
+	    -C client fetch-pack ../server \
+		$(git -C server rev-parse refs/heads/main)
+'
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd

--
2.39.2

[RESEND, BUG, SIGSEGV CRASH] (Re: [PATCH] fetch-pack: test: demonstrate segmentation fault when run with fsckObjects but without --lock-pack)

[Kirill Smelkov]

+ newren, peff, calvinwan, ps, avarab

Hello everyone. This patch demonstrates *segmentation fault* crash of Git
with, hopefully, properly written test. It was originally posted two
weeks ago without getting any replies nor any traces in seen/todo:

https://lore.kernel.org/git/20240606133605.602276-1-kirr@nexedi.com/T/#m9d454aadc8857b84f17d1a331739f7399cf1bbf8

I understand that there might be something wrong with my test, but could
you please at least provide some feedback and acknowledge the problem?

Resending the patch and adding more people to Cc who touched fetch-pack.c recently.

Thanks beforehand for feedback,
Kirill

---- 8< ----
From: Kirill Smelkov <kirr@nexedi.com>
Date: Thu, 6 Jun 2024 16:03:44 +0300
Subject: [PATCH] fetch-pack: test: demonstrate segmentation fault when run
 with fsckObjects but without --lock-pack
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

When running git-backup[1] over lab.nexedi.com archive we noticed that
Git started to crash on fetch-pack operation[2] after Git upgrade. We
tracked this down to be a regression introduced by Git 2.31, more
specifically by commit 5476e1efded5 (fetch-pack: print and use dangling
.gitmodules), which started to index and lock packfiles on do_keep=y &&
fsck_objects=y even if pack_lockfiles=NULL, which leads to
NULL-dereference when trying to append an entry to that pack_lockfiles
stringlist.

Please find attached a test that demonstrates this problem.

When run with test_expect_failure changed to test_expect_success the test
crashes with

    ./test-lib.sh: line 1063: 599675 Segmentation fault      (core dumped) git -c fetch.fsckObjects=true -c fetch.unpackLimit=1 -C client fetch-pack ../server $(git -C server rev-parse refs/heads/main)

and the backtrace is

    Program terminated with signal SIGSEGV, Segmentation fault.
    #0  0x0000558032c7ef3b in string_list_append_nodup (list=0x0,
        string=0x558033b695f0 ".git/objects/pack/pack-c1b2455a361bb50a0db087e7202db73d62938fa1.keep") at string-list.c:218
    218             ALLOC_GROW(list->items, list->nr + 1, list->alloc);
    (gdb) bt
    #0  0x0000558032c7ef3b in string_list_append_nodup (list=0x0,
        string=0x558033b695f0 ".git/objects/pack/pack-c1b2455a361bb50a0db087e7202db73d62938fa1.keep") at string-list.c:218
    #1  0x0000558032b5b83f in get_pack (args=0x7ffe680f3fa0, xd=0x7ffe680f4058, pack_lockfiles=0x0, index_pack_args=0x0,
        sought=0x558033b65e90, nr_sought=1, gitmodules_oids=0x558032e17b88 <fsck_options+72>) at fetch-pack.c:1042
    #2  0x0000558032b5e0b3 in do_fetch_pack_v2 (args=0x7ffe680f3fa0, fd=0x7ffe680f4058, orig_ref=0x558033b67b90, sought=0x558033b65e90,
        nr_sought=1, shallows=0x7ffe680f3df0, si=0x7ffe680f3e10, pack_lockfiles=0x0) at fetch-pack.c:1781
    #3  0x0000558032b5ef3c in fetch_pack (args=0x7ffe680f3fa0, fd=0x7ffe680f4058, ref=0x558033b67b90, sought=0x558033b65e90, nr_sought=1,
        shallow=0x7ffe680f3f80, pack_lockfiles=0x0, version=protocol_v2) at fetch-pack.c:2073
    #4  0x0000558032a08851 in cmd_fetch_pack (argc=3, argv=0x7ffe680f4500, prefix=0x0) at builtin/fetch-pack.c:242
    #5  0x00005580329b2be3 in run_builtin (p=0x558032e0bb30 <commands+1008>, argc=3, argv=0x7ffe680f4500) at git.c:474
    #6  0x00005580329b2ffe in handle_builtin (argc=3, argv=0x7ffe680f4500) at git.c:729
    #7  0x00005580329b3222 in run_argv (argcp=0x7ffe680f433c, argv=0x7ffe680f4330) at git.c:793
    #8  0x00005580329b3796 in cmd_main (argc=3, argv=0x7ffe680f4500) at git.c:928
    #9  0x0000558032ab9002 in main (argc=10, argv=0x7ffe680f44c8) at common-main.c:62

A simple debug patch below

    --- a/fetch-pack.c
    +++ b/fetch-pack.c
    @@ -1032,7 +1032,7 @@ static int get_pack(struct fetch_pack_args *args,
            cmd.git_cmd = 1;
            if (start_command(&cmd))
                    die(_("fetch-pack: unable to fork off %s"), cmd_name);
    -       if (do_keep && (pack_lockfiles || fsck_objects)) {
    +       if (do_keep && (pack_lockfiles /*|| fsck_objects*/)) {
                    int is_well_formed;
                    char *pack_lockfile = index_pack_lockfile(cmd.out, &is_well_formed);

makes the crash go away, but I did not investigated what should be the
proper logic.

Thanks beforehand for fixing this NULL-pointer dereference.

Kirill

[1] https://lab.nexedi.com/kirr/git-backup
[2] https://lab.nexedi.com/kirr/git-backup/-/blob/3230197c/git-backup.go#L717-739

Cc: Jonathan Tan <jonathantanmy@google.com>
Cc: Elijah Newren <newren@gmail.com>
Cc: Jeff King <peff@peff.net>
Cc: Calvin Wan <calvinwan@google.com>
Cc: Patrick Steinhardt <ps@pks.im>
Cc: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Cc: Alain Takoudjou <alain.takoudjou@nexedi.com>
Cc: Jérome Perrin <jerome@nexedi.com>
Signed-off-by: Kirill Smelkov <kirr@nexedi.com>
---
 t/t5500-fetch-pack.sh | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
index 1bc15a3f08..e3dbe79613 100755
--- a/t/t5500-fetch-pack.sh
+++ b/t/t5500-fetch-pack.sh
@@ -1056,6 +1056,20 @@ test_expect_success 'fetch with --filter=blob:limit=0' '
 	fetch_filter_blob_limit_zero server server
 '

+test_expect_failure 'fetch with fsckObjects but without --lock-pack does not segfault' '
+	rm -rf server client &&
+	git init server &&
+	test_commit -C server 1 &&
+
+	git init client &&
+	# unpackLimit=1 forces to keep received pack as pack instead of
+	# unpacking it to loose objects. The code is currently segfaulting when
+	# trying to index that kept pack.
+	git -c fetch.fsckObjects=true -c fetch.unpackLimit=1 \
+	    -C client fetch-pack ../server \
+		$(git -C server rev-parse refs/heads/main)
+'
+
 . "$TEST_DIRECTORY"/lib-httpd.sh
 start_httpd

--
2.39.2

---- 8< ----


On Thu, Jun 06, 2024 at 04:36:05PM +0300, Kirill Smelkov wrote:
> When running git-backup[1] over lab.nexedi.com archive we noticed that
> Git started to crash on fetch-pack operation[2] after Git upgrade. We
> tracked this down to be a regression introduced by Git 2.31, more
> specifically by commit 5476e1efded5 (fetch-pack: print and use dangling
> .gitmodules), which started to index and lock packfiles on do_keep=y &&
> fsck_objects=y even if pack_lockfiles=NULL, which leads to
> NULL-dereference when trying to append an entry to that pack_lockfiles
> stringlist.
>
> Please find attached a test that demonstrates this problem.
>
> When run with test_expect_failure changed to test_expect_success the test
> crashes with
>
>     ./test-lib.sh: line 1063: 599675 Segmentation fault      (core dumped) git -c fetch.fsckObjects=true -c fetch.unpackLimit=1 -C client fetch-pack ../server $(git -C server rev-parse refs/heads/main)
>
> and the backtrace is
>
>     Program terminated with signal SIGSEGV, Segmentation fault.
>     #0  0x0000558032c7ef3b in string_list_append_nodup (list=0x0,
>         string=0x558033b695f0 ".git/objects/pack/pack-c1b2455a361bb50a0db087e7202db73d62938fa1.keep") at string-list.c:218
>     218             ALLOC_GROW(list->items, list->nr + 1, list->alloc);
>     (gdb) bt
>     #0  0x0000558032c7ef3b in string_list_append_nodup (list=0x0,
>         string=0x558033b695f0 ".git/objects/pack/pack-c1b2455a361bb50a0db087e7202db73d62938fa1.keep") at string-list.c:218
>     #1  0x0000558032b5b83f in get_pack (args=0x7ffe680f3fa0, xd=0x7ffe680f4058, pack_lockfiles=0x0, index_pack_args=0x0,
>         sought=0x558033b65e90, nr_sought=1, gitmodules_oids=0x558032e17b88 <fsck_options+72>) at fetch-pack.c:1042
>     #2  0x0000558032b5e0b3 in do_fetch_pack_v2 (args=0x7ffe680f3fa0, fd=0x7ffe680f4058, orig_ref=0x558033b67b90, sought=0x558033b65e90,
>         nr_sought=1, shallows=0x7ffe680f3df0, si=0x7ffe680f3e10, pack_lockfiles=0x0) at fetch-pack.c:1781
>     #3  0x0000558032b5ef3c in fetch_pack (args=0x7ffe680f3fa0, fd=0x7ffe680f4058, ref=0x558033b67b90, sought=0x558033b65e90, nr_sought=1,
>         shallow=0x7ffe680f3f80, pack_lockfiles=0x0, version=protocol_v2) at fetch-pack.c:2073
>     #4  0x0000558032a08851 in cmd_fetch_pack (argc=3, argv=0x7ffe680f4500, prefix=0x0) at builtin/fetch-pack.c:242
>     #5  0x00005580329b2be3 in run_builtin (p=0x558032e0bb30 <commands+1008>, argc=3, argv=0x7ffe680f4500) at git.c:474
>     #6  0x00005580329b2ffe in handle_builtin (argc=3, argv=0x7ffe680f4500) at git.c:729
>     #7  0x00005580329b3222 in run_argv (argcp=0x7ffe680f433c, argv=0x7ffe680f4330) at git.c:793
>     #8  0x00005580329b3796 in cmd_main (argc=3, argv=0x7ffe680f4500) at git.c:928
>     #9  0x0000558032ab9002 in main (argc=10, argv=0x7ffe680f44c8) at common-main.c:62
>
> A simple debug patch below
>
>     --- a/fetch-pack.c
>     +++ b/fetch-pack.c
>     @@ -1032,7 +1032,7 @@ static int get_pack(struct fetch_pack_args *args,
>             cmd.git_cmd = 1;
>             if (start_command(&cmd))
>                     die(_("fetch-pack: unable to fork off %s"), cmd_name);
>     -       if (do_keep && (pack_lockfiles || fsck_objects)) {
>     +       if (do_keep && (pack_lockfiles /*|| fsck_objects*/)) {
>                     int is_well_formed;
>                     char *pack_lockfile = index_pack_lockfile(cmd.out, &is_well_formed);
>
> makes the crash go away, but I did not investigated what should be the
> proper logic.
>
> Thanks beforehand for fixing this NULL-pointer dereference.
>
> Kirill
>
> [1] https://lab.nexedi.com/kirr/git-backup
> [2] https://lab.nexedi.com/kirr/git-backup/-/blob/3230197c/git-backup.go#L717-739
>
> Cc: Jonathan Tan <jonathantanmy@google.com>
> Cc: Alain Takoudjou <alain.takoudjou@nexedi.com>
> Cc: Jérome Perrin <jerome@nexedi.com>
> Signed-off-by: Kirill Smelkov <kirr@nexedi.com>
> ---
>  t/t5500-fetch-pack.sh | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
>
> diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
> index 1bc15a3f08..e3dbe79613 100755
> --- a/t/t5500-fetch-pack.sh
> +++ b/t/t5500-fetch-pack.sh
> @@ -1056,6 +1056,20 @@ test_expect_success 'fetch with --filter=blob:limit=0' '
>  	fetch_filter_blob_limit_zero server server
>  '
>
> +test_expect_failure 'fetch with fsckObjects but without --lock-pack does not segfault' '
> +	rm -rf server client &&
> +	git init server &&
> +	test_commit -C server 1 &&
> +
> +	git init client &&
> +	# unpackLimit=1 forces to keep received pack as pack instead of
> +	# unpacking it to loose objects. The code is currently segfaulting when
> +	# trying to index that kept pack.
> +	git -c fetch.fsckObjects=true -c fetch.unpackLimit=1 \
> +	    -C client fetch-pack ../server \
> +		$(git -C server rev-parse refs/heads/main)
> +'
> +
>  . "$TEST_DIRECTORY"/lib-httpd.sh
>  start_httpd
>
> --
> 2.39.2

[PATCH] fetch-pack: fix segfault when fscking without --lock-pack

[Jeff King]

On Wed, Jun 19, 2024 at 05:14:21AM +0000, Kirill Smelkov wrote:

> + newren, peff, calvinwan, ps, avarab

It's fine (and even encouraged) to re-post a topic which didn't get any
attention the first time around. But please don't mass-cc unrelated
people like this. We can all read the list and see your re-post, and if
we don't respond it may be because we have other priorities.

> +test_expect_failure 'fetch with fsckObjects but without --lock-pack does not segfault' '
> +	rm -rf server client &&
> +	git init server &&
> +	test_commit -C server 1 &&
> +
> +	git init client &&
> +	# unpackLimit=1 forces to keep received pack as pack instead of
> +	# unpacking it to loose objects. The code is currently segfaulting when
> +	# trying to index that kept pack.
> +	git -c fetch.fsckObjects=true -c fetch.unpackLimit=1 \
> +	    -C client fetch-pack ../server \
> +		$(git -C server rev-parse refs/heads/main)
> +'

Thanks for providing a reproduction of the problem.

I think we don't want to stick the test right here, as it is breaking up
two related tests (though it is confusing because one uses http and the
other doesn't, so there's some http setup in between them). Though
curiously, putting the "rm -rf server" here revealed a minor bug in
those other tests. Fixed here:

  https://lore.kernel.org/git/20240619125255.GA346466@coredump.intra.peff.net

I think it's a bug that fetch.unpackLimit causes index-pack to create a
lockfile at all, but there's a more direct way to trigger the issue,
which I've used in the patch below. I'll follow up with more details in
a reply to the patch.

-- >8 --
Subject: fetch-pack: fix segfault when fscking without --lock-pack

The fetch-pack internals have multiple options related to creating
".keep" lock-files for the received pack:

  - if args.lock_pack is set, then we tell index-pack to create a .keep
    file. In the fetch-pack plumbing command, this is triggered by
    passing "-k" twice.

  - if the caller passes in a pack_lockfiles string list, then we use it
    to record the path of the keep-file created by index-pack. We get
    that name by reading the stdout of index-pack. In the fetch-pack
    command, this is triggered by passing the (undocumented) --lock-pack
    option; without it, we pass in a NULL string list.

So it's possible to ask index-pack to create the lock-file (using "-k
-k") but not ask to record it (by avoiding "--lock-pack"). This worked
fine until 5476e1efde (fetch-pack: print and use dangling .gitmodules,
2021-02-22), but now it causes a segfault.

Before that commit, if pack_lockfiles was NULL, we wouldn't bother
reading the output from index-pack at all. But since that commit,
index-pack may produce extra output if we asked it to fsck. So even if
nobody cares about the lockfile path, we still need to read it to skip
to the output we do care about.

We correctly check that we didn't get a NULL lockfile path (which can
happen if we did not ask it to create a .keep file at all), but we
missed the case where the lockfile path is not NULL (due to "-k -k") but
the pack_lockfiles string_list is NULL (because nobody passed
"--lock-pack"), and segfault trying to add to the NULL string-list.

We can fix this by skipping the append to the string list when either
the value or the list is NULL. In that case we must also free the
lockfile path to avoid leaking it when it's non-NULL.

Nobody noticed the bug for so long because the transport code used by
"git fetch" always passes in a pack_lockfiles pointer, and remote-curl
(the main user of the fetch-pack plumbing command) always passes
--lock-pack.

Reported-by: Kirill Smelkov <kirr@nexedi.com>
Signed-off-by: Jeff King <peff@peff.net>
---
 fetch-pack.c          |  4 +++-
 t/t5500-fetch-pack.sh | 10 ++++++++++
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/fetch-pack.c b/fetch-pack.c
index eba9e420ea..42f48fbc31 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -1038,8 +1038,10 @@ static int get_pack(struct fetch_pack_args *args,
 
 		if (!is_well_formed)
 			die(_("fetch-pack: invalid index-pack output"));
-		if (pack_lockfile)
+		if (pack_lockfiles && pack_lockfile)
 			string_list_append_nodup(pack_lockfiles, pack_lockfile);
+		else
+			free(pack_lockfile);
 		parse_gitmodules_oids(cmd.out, gitmodules_oids);
 		close(cmd.out);
 	}
diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
index b26f367620..585ea0ee16 100755
--- a/t/t5500-fetch-pack.sh
+++ b/t/t5500-fetch-pack.sh
@@ -993,6 +993,16 @@ test_expect_success 'ensure bogus fetch.negotiationAlgorithm yields error' '
 		       fetch origin server_has both_have_2
 '
 
+test_expect_success 'fetch-pack with fsckObjects and keep-file does not segfault' '
+	rm -rf server client &&
+	test_create_repo server &&
+	test_commit -C server one &&
+
+	test_create_repo client &&
+	git -c fetch.fsckObjects=true \
+	    -C client fetch-pack -k -k ../server HEAD
+'
+
 test_expect_success 'filtering by size' '
 	rm -rf server client &&
 	test_create_repo server &&
-- 
2.45.2.949.g1c649f6aed

Re: [PATCH] fetch-pack: fix segfault when fscking without --lock-pack

[Jeff King]

On Wed, Jun 19, 2024 at 09:02:56AM -0400, Jeff King wrote:

> I think it's a bug that fetch.unpackLimit causes index-pack to create a
> lockfile at all, but there's a more direct way to trigger the issue,
> which I've used in the patch below. I'll follow up with more details in
> a reply to the patch.

Your original test used transfer.unpackLimit. By itself that should just
cause us to avoid calling unpack-objects, keeping the pack we got, but
_not_ creating a .keep file. Likewise, if you pass one "-k" to
fetch-pack, it should just keep the pack but without a .keep file
(that's what the double "-k -k" does).

But both of these do trigger a .keep file, which seems wrong to me. The
caller has no idea that the .keep file was created, so it won't clean it
up, and the pack will be in limbo forever.

I tried bisecting and came up with fa74052922 (Always obtain fetch-pack
arguments from struct fetch_pack_args, 2007-09-19). Given the length of
time it's been this way, that makes me a little afraid to touch it. ;)
But I think in practice it is not really triggered because of what I
wrote earlier:

> Nobody noticed the bug for so long because the transport code used by
> "git fetch" always passes in a pack_lockfiles pointer, and remote-curl
> (the main user of the fetch-pack plumbing command) always passes
> --lock-pack.

That is, we're always asking for a lock-file anyway.

But it could affect external users of the fetch-pack plumbing. I.e., the
very command that produced the segfault for you is probably leaving an
unexpected .keep file in place.

> So it's possible to ask index-pack to create the lock-file (using "-k
> -k") but not ask to record it (by avoiding "--lock-pack"). This worked
> fine until 5476e1efde (fetch-pack: print and use dangling .gitmodules,
> 2021-02-22), but now it causes a segfault.
> 
> Before that commit, if pack_lockfiles was NULL, we wouldn't bother
> reading the output from index-pack at all. But since that commit,
> index-pack may produce extra output if we asked it to fsck. So even if
> nobody cares about the lockfile path, we still need to read it to skip
> to the output we do care about.

There's another interesting fallout from 5476e1efde that I noticed here.
Before that commit, if you did not pass --lock-pack to fetch-pack, then
we would never bother reading stdout from index-pack, and it would go to
the caller's stdout! So doing:

  git fetch-pack -k -k repo HEAD

would produce:

  keep	3282886e55735beb9a08b048394284b03bef8488

or similar on stdout. Which makes me wonder if some callers depend on
that. Or if it is simply a bug, since it would be intermingled with
fetch-pack's actual output. We still produce that output today, but if
you use fetch.fsckObjects, then we eat it while looking for the other
fsck-related output.

-Peff

Re: [PATCH] fetch-pack: fix segfault when fscking without --lock-pack

[Jeff King]

On Wed, Jun 19, 2024 at 09:22:08AM -0400, Jeff King wrote:

> On Wed, Jun 19, 2024 at 09:02:56AM -0400, Jeff King wrote:
> 
> > I think it's a bug that fetch.unpackLimit causes index-pack to create a
> > lockfile at all, but there's a more direct way to trigger the issue,
> > which I've used in the patch below. I'll follow up with more details in
> > a reply to the patch.
> 
> Your original test used transfer.unpackLimit. By itself that should just
> cause us to avoid calling unpack-objects, keeping the pack we got, but
> _not_ creating a .keep file. Likewise, if you pass one "-k" to
> fetch-pack, it should just keep the pack but without a .keep file
> (that's what the double "-k -k" does).
> 
> But both of these do trigger a .keep file, which seems wrong to me. The
> caller has no idea that the .keep file was created, so it won't clean it
> up, and the pack will be in limbo forever.
> 
> I tried bisecting and came up with fa74052922 (Always obtain fetch-pack
> arguments from struct fetch_pack_args, 2007-09-19). Given the length of
> time it's been this way, that makes me a little afraid to touch it. ;)

Even before that commit, we'd trigger the lock of unpack_limit was set
there. I find all of this code really puzzling (which makes me even more
hesitant to touch it). But I really don't understand why it is not just
this:

diff --git a/fetch-pack.c b/fetch-pack.c
index 42f48fbc31..ed57b0fdac 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -971,7 +971,7 @@ static int get_pack(struct fetch_pack_args *args,
 			strvec_push(&cmd.args, "-v");
 		if (args->use_thin_pack)
 			strvec_push(&cmd.args, "--fix-thin");
-		if ((do_keep || index_pack_args) && (args->lock_pack || unpack_limit))
+		if ((do_keep || index_pack_args) && args->lock_pack)
 			add_index_pack_keep_option(&cmd.args);
 		if (!index_pack_args && args->check_self_contained_and_connected)
 			strvec_push(&cmd.args, "--check-self-contained-and-connected");
diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
index 585ea0ee16..d6d6ea6281 100755
--- a/t/t5500-fetch-pack.sh
+++ b/t/t5500-fetch-pack.sh
@@ -1003,6 +1003,28 @@ test_expect_success 'fetch-pack with fsckObjects and keep-file does not segfault
 	    -C client fetch-pack -k -k ../server HEAD
 '
 
+test_expect_success 'fetch-pack -k does not create .keep file' '
+	rm -rf server client &&
+	test_create_repo server &&
+	test_commit -C server one &&
+
+	test_create_repo client &&
+	git -C client fetch-pack -k ../server HEAD &&
+	find client/.git/objects/pack -name "*.keep" >keep &&
+	test_must_be_empty keep
+'
+
+test_expect_success 'fetch-pack with unpackLimit does not create .keep file' '
+	rm -rf server client &&
+	test_create_repo server &&
+	test_commit -C server one &&
+
+	test_create_repo client &&
+	git -c fetch.unpackLimit=1 -C client fetch-pack ../server HEAD &&
+	find client/.git/objects/pack -name "*.keep" >keep &&
+	test_must_be_empty keep
+'
+
 test_expect_success 'filtering by size' '
 	rm -rf server client &&
 	test_create_repo server &&

Re: [PATCH] fetch-pack: fix segfault when fscking without --lock-pack

[Kirill Smelkov]

On Wed, Jun 19, 2024 at 09:02:56AM -0400, Jeff King wrote:
> On Wed, Jun 19, 2024 at 05:14:21AM +0000, Kirill Smelkov wrote:
> 
> > + newren, peff, calvinwan, ps, avarab
> 
> It's fine (and even encouraged) to re-post a topic which didn't get any
> attention the first time around. But please don't mass-cc unrelated
> people like this. We can all read the list and see your re-post, and if
> we don't respond it may be because we have other priorities.

Jeff, thanks for feedback and for reminding me about netiquette here.
Frankly I though that for some reason my report about segmentation fault
was classified as spam and did not get people attention due to that.

I appologize if my reposting with extended cc caused any inconvenience.


> > +test_expect_failure 'fetch with fsckObjects but without --lock-pack does not segfault' '
> > +	rm -rf server client &&
> > +	git init server &&
> > +	test_commit -C server 1 &&
> > +
> > +	git init client &&
> > +	# unpackLimit=1 forces to keep received pack as pack instead of
> > +	# unpacking it to loose objects. The code is currently segfaulting when
> > +	# trying to index that kept pack.
> > +	git -c fetch.fsckObjects=true -c fetch.unpackLimit=1 \
> > +	    -C client fetch-pack ../server \
> > +		$(git -C server rev-parse refs/heads/main)
> > +'
> 
> Thanks for providing a reproduction of the problem.
> 
> I think we don't want to stick the test right here, as it is breaking up
> two related tests (though it is confusing because one uses http and the
> other doesn't, so there's some http setup in between them). Though
> curiously, putting the "rm -rf server" here revealed a minor bug in
> those other tests. Fixed here:
> 
>   https://lore.kernel.org/git/20240619125255.GA346466@coredump.intra.peff.net
> 
> I think it's a bug that fetch.unpackLimit causes index-pack to create a
> lockfile at all, but there's a more direct way to trigger the issue,
> which I've used in the patch below. I'll follow up with more details in
> a reply to the patch.
> 
> -- >8 --
> Subject: fetch-pack: fix segfault when fscking without --lock-pack
> ...

Thanks for coming up with the fix promptly and for sure I might have
missed something when placing and doing my test. Thanks for correcting
that and coming with a more direct way to reproduce the issue.


On Wed, Jun 19, 2024 at 09:22:08AM -0400, Jeff King wrote:
> On Wed, Jun 19, 2024 at 09:02:56AM -0400, Jeff King wrote:
> 
> > I think it's a bug that fetch.unpackLimit causes index-pack to create a
> > lockfile at all, but there's a more direct way to trigger the issue,
> > which I've used in the patch below. I'll follow up with more details in
> > a reply to the patch.
> 
> Your original test used transfer.unpackLimit. By itself that should just
> cause us to avoid calling unpack-objects, keeping the pack we got, but
> _not_ creating a .keep file. Likewise, if you pass one "-k" to
> fetch-pack, it should just keep the pack but without a .keep file
> (that's what the double "-k -k" does).
> 
> But both of these do trigger a .keep file, which seems wrong to me. The
> caller has no idea that the .keep file was created, so it won't clean it
> up, and the pack will be in limbo forever.
> 
> I tried bisecting and came up with fa74052922 (Always obtain fetch-pack
> arguments from struct fetch_pack_args, 2007-09-19). Given the length of
> time it's been this way, that makes me a little afraid to touch it. ;)
> But I think in practice it is not really triggered because of what I
> wrote earlier:
> 
> > Nobody noticed the bug for so long because the transport code used by
> > "git fetch" always passes in a pack_lockfiles pointer, and remote-curl
> > (the main user of the fetch-pack plumbing command) always passes
> > --lock-pack.
> 
> That is, we're always asking for a lock-file anyway.
> 
> But it could affect external users of the fetch-pack plumbing. I.e., the
> very command that produced the segfault for you is probably leaving an
> unexpected .keep file in place.

Thanks for noticing. Yes, indeed I use `git fetch-pack` and expect only
the pack to be fetched without remaining it in pinned state via .keep file.

It would be good if that could be fixed because present behaviour
contradicts fetch-pack documentation that says that the pack is locked
against being repacked only on double --keep.


> > So it's possible to ask index-pack to create the lock-file (using "-k
> > -k") but not ask to record it (by avoiding "--lock-pack"). This worked
> > fine until 5476e1efde (fetch-pack: print and use dangling .gitmodules,
> > 2021-02-22), but now it causes a segfault.
> > 
> > Before that commit, if pack_lockfiles was NULL, we wouldn't bother
> > reading the output from index-pack at all. But since that commit,
> > index-pack may produce extra output if we asked it to fsck. So even if
> > nobody cares about the lockfile path, we still need to read it to skip
> > to the output we do care about.
> 
> There's another interesting fallout from 5476e1efde that I noticed here.
> Before that commit, if you did not pass --lock-pack to fetch-pack, then
> we would never bother reading stdout from index-pack, and it would go to
> the caller's stdout! So doing:
> 
>   git fetch-pack -k -k repo HEAD
> 
> would produce:
> 
>   keep	3282886e55735beb9a08b048394284b03bef8488
> 
> or similar on stdout. Which makes me wonder if some callers depend on
> that. Or if it is simply a bug, since it would be intermingled with
> fetch-pack's actual output. We still produce that output today, but if
> you use fetch.fsckObjects, then we eat it while looking for the other
> fsck-related output.

Thanks for noticing as well. Indeed that lines emitted with keep might
be wanted to consume by a plumbing user. I'm not consuming them myself,
but from the output it looks like to be intended. Maybe almost noone
depend on those this days though.

For the reference: on my side I switched to fetch-pack plumbing from
fetch porcelain because `git fetch` was working really slow when
initially checking whether it already locally has all the objects
advertised by remote:

https://lab.nexedi.com/kirr/git-backup/commit/899103bf
https://lab.nexedi.com/kirr/git-backup/commit/3efed898


On Wed, Jun 19, 2024 at 09:35:22AM -0400, Jeff King wrote:
> On Wed, Jun 19, 2024 at 09:22:08AM -0400, Jeff King wrote:
> 
> > On Wed, Jun 19, 2024 at 09:02:56AM -0400, Jeff King wrote:
> > 
> > > I think it's a bug that fetch.unpackLimit causes index-pack to create a
> > > lockfile at all, but there's a more direct way to trigger the issue,
> > > which I've used in the patch below. I'll follow up with more details in
> > > a reply to the patch.
> > 
> > Your original test used transfer.unpackLimit. By itself that should just
> > cause us to avoid calling unpack-objects, keeping the pack we got, but
> > _not_ creating a .keep file. Likewise, if you pass one "-k" to
> > fetch-pack, it should just keep the pack but without a .keep file
> > (that's what the double "-k -k" does).
> > 
> > But both of these do trigger a .keep file, which seems wrong to me. The
> > caller has no idea that the .keep file was created, so it won't clean it
> > up, and the pack will be in limbo forever.
> > 
> > I tried bisecting and came up with fa74052922 (Always obtain fetch-pack
> > arguments from struct fetch_pack_args, 2007-09-19). Given the length of
> > time it's been this way, that makes me a little afraid to touch it. ;)
> 
> Even before that commit, we'd trigger the lock of unpack_limit was set
> there. I find all of this code really puzzling (which makes me even more
> hesitant to touch it). But I really don't understand why it is not just
> this:
> 
> diff --git a/fetch-pack.c b/fetch-pack.c
> index 42f48fbc31..ed57b0fdac 100644
> --- a/fetch-pack.c
> +++ b/fetch-pack.c
> @@ -971,7 +971,7 @@ static int get_pack(struct fetch_pack_args *args,
>  			strvec_push(&cmd.args, "-v");
>  		if (args->use_thin_pack)
>  			strvec_push(&cmd.args, "--fix-thin");
> -		if ((do_keep || index_pack_args) && (args->lock_pack || unpack_limit))
> +		if ((do_keep || index_pack_args) && args->lock_pack)
>  			add_index_pack_keep_option(&cmd.args);
>  		if (!index_pack_args && args->check_self_contained_and_connected)
>  			strvec_push(&cmd.args, "--check-self-contained-and-connected");
> diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
> index 585ea0ee16..d6d6ea6281 100755
> --- a/t/t5500-fetch-pack.sh
> +++ b/t/t5500-fetch-pack.sh
> @@ -1003,6 +1003,28 @@ test_expect_success 'fetch-pack with fsckObjects and keep-file does not segfault
>  	    -C client fetch-pack -k -k ../server HEAD
>  '
>  
> +test_expect_success 'fetch-pack -k does not create .keep file' '
> +	rm -rf server client &&
> +	test_create_repo server &&
> +	test_commit -C server one &&
> +
> +	test_create_repo client &&
> +	git -C client fetch-pack -k ../server HEAD &&
> +	find client/.git/objects/pack -name "*.keep" >keep &&
> +	test_must_be_empty keep
> +'
> +
> +test_expect_success 'fetch-pack with unpackLimit does not create .keep file' '
> +	rm -rf server client &&
> +	test_create_repo server &&
> +	test_commit -C server one &&
> +
> +	test_create_repo client &&
> +	git -c fetch.unpackLimit=1 -C client fetch-pack ../server HEAD &&
> +	find client/.git/objects/pack -name "*.keep" >keep &&
> +	test_must_be_empty keep
> +'
> +
>  test_expect_success 'filtering by size' '
>  	rm -rf server client &&
>  	test_create_repo server &&

I once worked on Git internals in 72441af7c4e3 and around, but that was
for tree-diff and today I focus primarily on other areas. So I trust
your best judgement, and best judgement of people who are currently
active in Git development, to see whether it is ok or not.

On my side, as a user and primarily looking at the tests, it looks like
to be a good thing to do not to create an unexpected .keep file.

Thanks, once again, for your prompt reply, explanations and for the fixes.

Kirill

Re: [PATCH] fetch-pack: fix segfault when fscking without --lock-pack

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Before that commit, if pack_lockfiles was NULL, we wouldn't bother
> reading the output from index-pack at all. But since that commit,
> index-pack may produce extra output if we asked it to fsck. So even if
> nobody cares about the lockfile path, we still need to read it to skip
> to the output we do care about.

True.  Looking at that problematic commit, I wonder how it passed
the review process.  As a design, adding a list of bare object IDs
without marking what they are for is way too inextensible by our
standard practice.

It is probably not too late to fix it, as this is purely an internal
implementation detail between fetch-pack and index-pack that is not
even documented ("git index-pack --help" does talk about the
"(pack|keep)\t<pack-name>" output, but never the output after that).

> We correctly check that we didn't get a NULL lockfile path (which can
> happen if we did not ask it to create a .keep file at all), but we
> missed the case where the lockfile path is not NULL (due to "-k -k") but
> the pack_lockfiles string_list is NULL (because nobody passed
> "--lock-pack"), and segfault trying to add to the NULL string-list.
>
> We can fix this by skipping the append to the string list when either
> the value or the list is NULL. In that case we must also free the
> lockfile path to avoid leaking it when it's non-NULL.

OK.

> diff --git a/fetch-pack.c b/fetch-pack.c
> index eba9e420ea..42f48fbc31 100644
> --- a/fetch-pack.c
> +++ b/fetch-pack.c
> @@ -1038,8 +1038,10 @@ static int get_pack(struct fetch_pack_args *args,
>  
>  		if (!is_well_formed)
>  			die(_("fetch-pack: invalid index-pack output"));
> -		if (pack_lockfile)
> +		if (pack_lockfiles && pack_lockfile)
>  			string_list_append_nodup(pack_lockfiles, pack_lockfile);
> +		else
> +			free(pack_lockfile);
>  		parse_gitmodules_oids(cmd.out, gitmodules_oids);
>  		close(cmd.out);
>  	}

That looks like a very safe thing to do.

> diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
> index b26f367620..585ea0ee16 100755
> --- a/t/t5500-fetch-pack.sh
> +++ b/t/t5500-fetch-pack.sh
> @@ -993,6 +993,16 @@ test_expect_success 'ensure bogus fetch.negotiationAlgorithm yields error' '
>  		       fetch origin server_has both_have_2
>  '
>  
> +test_expect_success 'fetch-pack with fsckObjects and keep-file does not segfault' '
> +	rm -rf server client &&
> +	test_create_repo server &&
> +	test_commit -C server one &&
> +
> +	test_create_repo client &&
> +	git -c fetch.fsckObjects=true \
> +	    -C client fetch-pack -k -k ../server HEAD
> +'
> +

And the test is quite straight-forward.

Will queue.  Thanks.