* [PATCH v4 1/9] refs: remove unused typedef 'ref_transaction_commit_fn'
From: Karthik Nayak @ 2026-05-04 17:44 UTC (permalink / raw)
To: git; +Cc: ps, toon, Karthik Nayak
In-Reply-To: <20260504-refs-move-to-generic-layer-v4-0-936ac2f0b1a3@gmail.com>
The typedef 'ref_transaction_commit_fn' is not used anywhere in our
code, let's remove it.
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
refs/refs-internal.h | 4 ----
1 file changed, 4 deletions(-)
diff --git a/refs/refs-internal.h b/refs/refs-internal.h
index d79e35fd26..2d963cc4f4 100644
--- a/refs/refs-internal.h
+++ b/refs/refs-internal.h
@@ -421,10 +421,6 @@ typedef int ref_transaction_abort_fn(struct ref_store *refs,
struct ref_transaction *transaction,
struct strbuf *err);
-typedef int ref_transaction_commit_fn(struct ref_store *refs,
- struct ref_transaction *transaction,
- struct strbuf *err);
-
typedef int optimize_fn(struct ref_store *ref_store,
struct refs_optimize_opts *opts);
--
2.53.GIT
^ permalink raw reply related
* [PATCH v4 0/9] refs: move some of the generic logic out of the backends
From: Karthik Nayak @ 2026-05-04 17:44 UTC (permalink / raw)
To: git; +Cc: ps, toon, Karthik Nayak
In-Reply-To: <20260420-refs-move-to-generic-layer-v1-0-513e354f376b@gmail.com>
This series came together while I was working on other reference related
code and realized that some of the individual logic implemented with the
reference backends can be moved to the generic layer.
Moving code to the generic layer, simplifies the responsibility of
individual backends and avoids deviation in logic between the backends.
The biggest changes are related to moving out usage of `parse_object()`
and `peel_object()` from reference transactions. The former is used to
validate that the OID provided points to a commit object. The latter is
an optimization technique where the packed/reftable backend store the
peeled OID whenever available, so reading such references provides the
peeled OID without having to call the ODB.
Moving object parsing to the generic layout involves moving it out of
the prepare stage of the transaction and into `ref_transaction_update()`
where every added update is checked. As such, this also involves
modifying update-ref(1) and receive-pack(1) to follow this paradigm.
---
Changes in v4:
- Fix a bug in the error handling code, move it to a function and also
add missing call sites. Add tests to ensure we catch this.
- Link to v3: https://patch.msgid.link/20260427-refs-move-to-generic-layer-v3-0-e4638dfb7897@gmail.com
Changes in v3:
- Remove an unwanted change which creeped up during a rebase.
- Add information in the commit message around how the order of errors
in git-update-ref(1) will change while maintaining functionality.
- Change up the order of an `if..else` to make it clearer.
- Other small typos and fixes.
- Link to v2: https://patch.msgid.link/20260423-refs-move-to-generic-layer-v2-0-ae5a4f146d7d@gmail.com
Changes in v2:
- Split the second commit into two: one introducing
`ref_store_init_options` and the second to use it for reflog config.
- Use opts as the variable name consistently.
- A bunch of grammar fixes.
- Link to v1: https://patch.msgid.link/20260420-refs-move-to-generic-layer-v1-0-513e354f376b@gmail.com
---
builtin/receive-pack.c | 22 +++---
builtin/update-ref.c | 182 +++++++++++++++++++++++++++++++-----------------
refs.c | 60 ++++++++++++----
refs.h | 16 ++---
refs/files-backend.c | 58 ++++++---------
refs/packed-backend.c | 10 ++-
refs/packed-backend.h | 3 +-
refs/refs-internal.h | 35 ++++++++--
refs/reftable-backend.c | 40 +++--------
t/t1400-update-ref.sh | 14 ++++
10 files changed, 266 insertions(+), 174 deletions(-)
Karthik Nayak (9):
refs: remove unused typedef 'ref_transaction_commit_fn'
refs: introduce `ref_store_init_options`
refs: extract out reflog config to generic layer
refs: return `ref_transaction_error` from `ref_transaction_update()`
update-ref: move `print_rejected_refs()` up
update-ref: handle rejections while adding updates
refs: move object parsing to the generic layer
refs: add peeled object ID to the `ref_update` struct
refs: use peeled tag values in reference backends
Range-diff versus v3:
1: b89e273583 = 1: cf5fa53f34 refs: remove unused typedef 'ref_transaction_commit_fn'
2: 8f7f8dadc8 = 2: afd10fd7a3 refs: introduce `ref_store_init_options`
3: 9cdc35f5b6 = 3: 4eb6950cdc refs: extract out reflog config to generic layer
4: 36c0c86a31 = 4: dd8177a243 refs: return `ref_transaction_error` from `ref_transaction_update()`
5: 1b7eca9353 = 5: eb86178ae5 update-ref: move `print_rejected_refs()` up
6: aef1529054 ! 6: 464c6371a4 update-ref: handle rejections while adding updates
@@ builtin/update-ref.c: static unsigned int default_flags;
/*
* Parse one whitespace- or NUL-terminated, possibly C-quoted argument
* and append the result to arg. Return a pointer to the terminator.
+@@ builtin/update-ref.c: static void print_rejected_refs(const char *refname,
+ strbuf_release(&sb);
+ }
+
++/*
++ * Handle transaction errors. If we're using batches updates, we want to only
++ * die for generic errors and print the remaining to the user.
++ */
++static void handle_ref_transaction_error(const char *refname,
++ struct object_id *new_oid,
++ struct object_id *old_oid,
++ const char *new_target,
++ const char *old_target,
++ enum ref_transaction_error tx_err,
++ struct strbuf *err,
++ struct command_options *opts)
++{
++ if (!tx_err)
++ return;
++
++ if (tx_err != REF_TRANSACTION_ERROR_GENERIC && opts->allow_update_failures) {
++ print_rejected_refs(refname, old_oid, new_oid, old_target,
++ new_target, tx_err, err->buf, NULL);
++ return;
++ }
++
++ die("%s", err->buf);
++}
++
+ /*
+ * The following five parse_cmd_*() functions parse the corresponding
+ * command. In each case, next points at the character following the
@@ builtin/update-ref.c: static void print_rejected_refs(const char *refname,
*/
@@ builtin/update-ref.c: static void parse_cmd_update(struct ref_transaction *trans
- NULL, NULL,
- update_flags | create_reflog_flag,
- msg, &err))
+- die("%s", err.buf);
+ tx_err = ref_transaction_update(transaction, refname,
+ &new_oid, have_old ? &old_oid : NULL,
+ NULL, NULL,
+ update_flags | create_reflog_flag,
+ msg, &err);
++ handle_ref_transaction_error(refname, &new_oid, have_old ? &old_oid : NULL,
++ NULL, NULL, tx_err, &err, opts);
+
-+ /*
-+ * Generic errors are non-recoverable, so we cannot skip the update
-+ * or mark it as rejected.
-+ */
-+ if (tx_err == REF_TRANSACTION_ERROR_GENERIC)
- die("%s", err.buf);
-+ if (tx_err && opts->allow_update_failures)
-+ print_rejected_refs(refname, have_old ? &old_oid : NULL,
-+ &new_oid, NULL, NULL, tx_err, err.buf,
-+ NULL);
-+
update_flags = default_flags;
free(refname);
- strbuf_release(&err);
+@@ builtin/update-ref.c: static void parse_cmd_update(struct ref_transaction *transaction,
}
static void parse_cmd_symref_update(struct ref_transaction *transaction,
@@ builtin/update-ref.c: static void parse_cmd_symref_update(struct ref_transaction
- have_old_oid ? NULL : old_target,
- update_flags | create_reflog_flag,
- msg, &err))
+- die("%s", err.buf);
+ tx_err = ref_transaction_update(transaction, refname, NULL,
+ have_old_oid ? &old_oid : NULL,
+ new_target,
+ have_old_oid ? NULL : old_target,
+ update_flags | create_reflog_flag,
+ msg, &err);
-+
-+ /*
-+ * Generic errors are non-recoverable, so we cannot skip the update
-+ * or mark it as rejected.
-+ */
-+ if (tx_err == REF_TRANSACTION_ERROR_GENERIC)
- die("%s", err.buf);
++ handle_ref_transaction_error(refname, NULL, have_old_oid ? &old_oid : NULL,
++ new_target, have_old_oid ? NULL : old_target,
++ tx_err, &err, opts);
-+ if (tx_err && opts->allow_update_failures)
-+ print_rejected_refs(refname, have_old_oid ? &old_oid : NULL,
-+ NULL, have_old_oid ? NULL : old_target,
-+ new_target, tx_err, err.buf, NULL);
-+
update_flags = default_flags;
free(refname);
- free(old_arg);
@@ builtin/update-ref.c: static void parse_cmd_symref_update(struct ref_transaction *transaction,
}
static void parse_cmd_create(struct ref_transaction *transaction,
- const char *next, const char *end)
+ const char *next, const char *end,
-+ struct command_options *opts UNUSED)
++ struct command_options *opts)
{
struct strbuf err = STRBUF_INIT;
char *refname;
+ struct object_id new_oid;
++ enum ref_transaction_error tx_err;
+
+ refname = parse_refname(&next);
+ if (!refname)
@@ builtin/update-ref.c: static void parse_cmd_create(struct ref_transaction *transaction,
+ if (*next != line_termination)
+ die("create %s: extra input: %s", refname, next);
+
+- if (ref_transaction_create(transaction, refname, &new_oid, NULL,
+- update_flags | create_reflog_flag,
+- msg, &err))
+- die("%s", err.buf);
++ tx_err = ref_transaction_create(transaction, refname, &new_oid, NULL,
++ update_flags | create_reflog_flag,
++ msg, &err);
++ handle_ref_transaction_error(refname, &new_oid, NULL, NULL, NULL, tx_err,
++ &err, opts);
+
+ update_flags = default_flags;
+ free(refname);
strbuf_release(&err);
}
@@ builtin/update-ref.c: static void parse_cmd_create(struct ref_transaction *trans
static void parse_cmd_symref_create(struct ref_transaction *transaction,
- const char *next, const char *end UNUSED)
+ const char *next, const char *end UNUSED,
-+ struct command_options *opts UNUSED)
++ struct command_options *opts)
{
struct strbuf err = STRBUF_INIT;
char *refname, *new_target;
++ enum ref_transaction_error tx_err;
+
+ refname = parse_refname(&next);
+ if (!refname)
+@@ builtin/update-ref.c: static void parse_cmd_symref_create(struct ref_transaction *transaction,
+ if (*next != line_termination)
+ die("symref-create %s: extra input: %s", refname, next);
+
+- if (ref_transaction_create(transaction, refname, NULL, new_target,
+- update_flags | create_reflog_flag,
+- msg, &err))
+- die("%s", err.buf);
++ tx_err = ref_transaction_create(transaction, refname, NULL, new_target,
++ update_flags | create_reflog_flag,
++ msg, &err);
++ handle_ref_transaction_error(refname, NULL, NULL, new_target, NULL,
++ tx_err, &err, opts);
+
+ update_flags = default_flags;
+ free(refname);
@@ builtin/update-ref.c: static void parse_cmd_symref_create(struct ref_transaction *transaction,
}
7: 4b225f4f4d ! 7: f971226d89 refs: move object parsing to the generic layer
@@ refs/reftable-backend.c: static enum ref_transaction_error prepare_single_update
/*
* When we update the reference that HEAD points to we enqueue
* a second log-only update for HEAD so that its reflog is
+
+ ## t/t1400-update-ref.sh ##
+@@ t/t1400-update-ref.sh: test_expect_success 'stdin -z create ref fails with empty new value' '
+ test_must_fail git rev-parse --verify -q $c
+ '
+
++test_expect_success 'stdin -z create ref fails with non commit object' '
++ printf $F "create $c" "$(test_oid 001)" >stdin &&
++ test_must_fail git update-ref -z --stdin <stdin 2>err &&
++ grep "fatal: trying to write ref ${SQ}$c${SQ} with nonexistent object" err &&
++ test_must_fail git rev-parse --verify -q $c
++'
++
++test_expect_success 'stdin -z update ref fails with non commit object' '
++ printf $F "update $b" "$(test_oid 001)" "" >stdin &&
++ test_must_fail git update-ref -z --stdin <stdin 2>err &&
++ grep "fatal: trying to write ref ${SQ}$b${SQ} with nonexistent object" err &&
++ test_must_fail git rev-parse --verify -q $c
++'
++
+ test_expect_success 'stdin -z update ref works with right old value' '
+ printf $F "update $b" "$m~1" "$m" >stdin &&
+ git update-ref -z --stdin <stdin &&
8: 944af6a454 = 8: d271167077 refs: add peeled object ID to the `ref_update` struct
9: 47653fdde9 = 9: ed88b9e2ce refs: use peeled tag values in reference backends
base-commit: f65aba1e87db64413b6d1ed5ae5a45b5a84a0997
change-id: 20260417-refs-move-to-generic-layer-f7525c5e8764
Thanks
- Karthik
^ permalink raw reply
* [PATCH v2 11/11] ci: run expensive tests on push builds to integration branches
From: Johannes Schindelin via GitGitGadget @ 2026-05-04 17:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Torsten Bögershausen, Jeff King,
Johannes Schindelin, Johannes Schindelin
In-Reply-To: <pull.2102.v2.git.1777914508.gitgitgadget@gmail.com>
From: Johannes Schindelin <johannes.schindelin@gmx.de>
Derrick Stolee suggested [1] that expensive tests should be run at a
regular cadence rather than on every PR iteration. Gate GIT_TEST_LONG
on push builds to the integration branches (next, master, main, maint)
so that the EXPENSIVE prereq is satisfied there but not during PR
validation, where the extra minutes of wall-clock time do not justify
themselves.
[1] https://lore.kernel.org/git/e1e8837f-7374-4079-ba87-ab95dd156e33@gmail.com/
Helped-by: Derrick Stolee <derrickstolee@github.com>
Assisted-by: Claude Opus 4.6
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
ci/lib.sh | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/ci/lib.sh b/ci/lib.sh
index 42a2b6a318..a671994bdf 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -314,6 +314,15 @@ export DEFAULT_TEST_TARGET=prove
export GIT_TEST_CLONE_2GB=true
export SKIP_DASHED_BUILT_INS=YesPlease
+# Enable expensive tests on push builds to integration branches, but
+# not on PR builds where the extra time is not justified for every
+# iteration.
+case "$GITHUB_EVENT_NAME,$CI_BRANCH" in
+push,*next*|push,*master*|push,*main*|push,*maint*)
+ export GIT_TEST_LONG=YesPlease
+ ;;
+esac
+
case "$distro" in
ubuntu-*)
# Python 2 is end of life, and Ubuntu 23.04 and newer don't actually
--
gitgitgadget
^ permalink raw reply related
* [PATCH v2 10/11] t5608: mark >4GB tests as EXPENSIVE
From: Johannes Schindelin via GitGitGadget @ 2026-05-04 17:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Torsten Bögershausen, Jeff King,
Johannes Schindelin, Johannes Schindelin
In-Reply-To: <pull.2102.v2.git.1777914508.gitgitgadget@gmail.com>
From: Johannes Schindelin <johannes.schindelin@gmx.de>
Even with precomputed pack constants that reduced the helper's
runtime from minutes to seconds, the >4GB clone tests still take
200-850 seconds across CI jobs. The bottleneck is no longer the
pack generation but the clone operations themselves: transporting,
unpacking, and indexing 4 GiB of data through unpack-objects and
index-pack is inherently expensive.
As Jeff King pointed out [1], t5608 alone takes 160 seconds on his
laptop while the rest of the entire test suite finishes in under 90
seconds, and the test's disk footprint (4+ GiB source repo, then
two clones) is problematic for developers who use RAM disks for
their trash directories.
Gate the >4GB tests on the EXPENSIVE prereq (which requires
GIT_TEST_LONG to be set) in addition to SIZE_T_IS_64BIT, keeping
them out of normal local test runs.
[1] https://lore.kernel.org/git/20260501063805.GA2038915@coredump.intra.peff.net/
Assisted-by: Claude Opus 4.6
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
t/t5608-clone-2gb.sh | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/t/t5608-clone-2gb.sh b/t/t5608-clone-2gb.sh
index af93302dde..4f8a95ddda 100755
--- a/t/t5608-clone-2gb.sh
+++ b/t/t5608-clone-2gb.sh
@@ -49,7 +49,7 @@ test_expect_success 'clone - with worktree, file:// protocol' '
'
-test_expect_success SIZE_T_IS_64BIT 'set up repo with >4GB object' '
+test_expect_success SIZE_T_IS_64BIT,EXPENSIVE 'set up repo with >4GB object' '
large_blob_size=$((4*1024*1024*1024+1)) &&
git init --bare 4gb-repo &&
head_oid=$(test-tool synthesize pack \
@@ -60,7 +60,7 @@ test_expect_success SIZE_T_IS_64BIT 'set up repo with >4GB object' '
git -C 4gb-repo symbolic-ref HEAD refs/heads/main
'
-test_expect_success SIZE_T_IS_64BIT 'clone >4GB object via unpack-objects' '
+test_expect_success SIZE_T_IS_64BIT,EXPENSIVE 'clone >4GB object via unpack-objects' '
# The synthesized pack has five objects, so a large unpack limit keeps
# fetch-pack on the unpack-objects path.
git -c fetch.unpackLimit=100 clone --bare \
@@ -76,7 +76,7 @@ test_expect_success SIZE_T_IS_64BIT 'clone >4GB object via unpack-objects' '
test "$source_blob" = "$clone_blob"
'
-test_expect_success SIZE_T_IS_64BIT 'clone with >4GB object via index-pack' '
+test_expect_success SIZE_T_IS_64BIT,EXPENSIVE 'clone with >4GB object via index-pack' '
# Force fetch-pack to hand the pack to index-pack instead.
git -c fetch.unpackLimit=1 clone --bare \
"file://$(pwd)/4gb-repo" 4gb-clone-index &&
--
gitgitgadget
^ permalink raw reply related
* [PATCH v2 09/11] test-tool synthesize: add precomputed SHA-256 pack for 4 GiB + 1
From: Johannes Schindelin via GitGitGadget @ 2026-05-04 17:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Torsten Bögershausen, Jeff King,
Johannes Schindelin, Johannes Schindelin
In-Reply-To: <pull.2102.v2.git.1777914508.gitgitgadget@gmail.com>
From: Johannes Schindelin <johannes.schindelin@gmx.de>
Add a SHA-256 entry to the fast_packs[] table. The pack prefix and
deflate block structure are identical to SHA-1 (the pack format does
not encode the hash algorithm in its header). Only the suffix differs:
SHA-256 OIDs are 32 bytes instead of 20, giving a 609-byte suffix
compared to 513 for SHA-1, and a different pack checksum.
The constants were generated by running the generic path inside a
repository initialized with --object-format=sha256.
Assisted-by: Claude Opus 4.6
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
t/helper/test-synthesize.c | 91 ++++++++++++++++++++++++++++++++++++++
1 file changed, 91 insertions(+)
diff --git a/t/helper/test-synthesize.c b/t/helper/test-synthesize.c
index 83c40ee02a..1f28ecf0f2 100644
--- a/t/helper/test-synthesize.c
+++ b/t/helper/test-synthesize.c
@@ -246,6 +246,90 @@ static const unsigned char fast_pack_sha1_suffix[] = {
0xe3
};
+/*
+ * SHA-256 suffix: same structure, but with 32-byte OIDs and SHA-256
+ * pack checksum (609 bytes vs 513 for SHA-1).
+ */
+static const unsigned char fast_pack_sha256_suffix[] = {
+ 0xac, 0x02, 0x78, 0x01, 0x01, 0x2c, 0x00, 0xd3,
+ 0xff, 0x31, 0x30, 0x30, 0x36, 0x34, 0x34, 0x20,
+ 0x66, 0x69, 0x6c, 0x65, 0x00, 0x42, 0x53, 0xc1,
+ 0x8a, 0x9f, 0x5e, 0xc3, 0xbb, 0x47, 0xb0, 0x83,
+ 0x8a, 0x19, 0xdb, 0x31, 0xbb, 0x7b, 0x0f, 0x3b,
+ 0x80, 0xa4, 0xbc, 0x2f, 0xaf, 0x72, 0x6b, 0xdb,
+ 0x62, 0xaa, 0xba, 0xdd, 0xde, 0x77, 0xc6, 0x13,
+ 0xeb, 0x9d, 0x0c, 0x78, 0x01, 0x01, 0xcd, 0x00,
+ 0x32, 0xff, 0x74, 0x72, 0x65, 0x65, 0x20, 0x62,
+ 0x36, 0x30, 0x39, 0x37, 0x37, 0x64, 0x37, 0x63,
+ 0x34, 0x63, 0x32, 0x64, 0x31, 0x65, 0x63, 0x63,
+ 0x33, 0x66, 0x62, 0x61, 0x31, 0x64, 0x39, 0x38,
+ 0x65, 0x65, 0x31, 0x32, 0x30, 0x61, 0x64, 0x63,
+ 0x32, 0x34, 0x38, 0x33, 0x34, 0x39, 0x35, 0x30,
+ 0x62, 0x65, 0x34, 0x31, 0x32, 0x64, 0x39, 0x34,
+ 0x63, 0x38, 0x30, 0x39, 0x34, 0x38, 0x30, 0x66,
+ 0x35, 0x38, 0x62, 0x61, 0x39, 0x64, 0x61, 0x0a,
+ 0x61, 0x75, 0x74, 0x68, 0x6f, 0x72, 0x20, 0x41,
+ 0x20, 0x55, 0x20, 0x54, 0x68, 0x6f, 0x72, 0x20,
+ 0x3c, 0x61, 0x75, 0x74, 0x68, 0x6f, 0x72, 0x40,
+ 0x65, 0x78, 0x61, 0x6d, 0x70, 0x6c, 0x65, 0x2e,
+ 0x63, 0x6f, 0x6d, 0x3e, 0x20, 0x31, 0x32, 0x33,
+ 0x34, 0x35, 0x36, 0x37, 0x38, 0x39, 0x30, 0x20,
+ 0x2b, 0x30, 0x30, 0x30, 0x30, 0x0a, 0x63, 0x6f,
+ 0x6d, 0x6d, 0x69, 0x74, 0x74, 0x65, 0x72, 0x20,
+ 0x43, 0x20, 0x4f, 0x20, 0x4d, 0x69, 0x74, 0x74,
+ 0x65, 0x72, 0x20, 0x3c, 0x63, 0x6f, 0x6d, 0x6d,
+ 0x69, 0x74, 0x74, 0x65, 0x72, 0x40, 0x65, 0x78,
+ 0x61, 0x6d, 0x70, 0x6c, 0x65, 0x2e, 0x63, 0x6f,
+ 0x6d, 0x3e, 0x20, 0x31, 0x32, 0x33, 0x34, 0x35,
+ 0x36, 0x37, 0x38, 0x39, 0x30, 0x20, 0x2b, 0x30,
+ 0x30, 0x30, 0x30, 0x0a, 0x0a, 0x4c, 0x61, 0x72,
+ 0x67, 0x65, 0x20, 0x62, 0x6c, 0x6f, 0x62, 0x20,
+ 0x63, 0x6f, 0x6d, 0x6d, 0x69, 0x74, 0x0a, 0xb7,
+ 0x80, 0x3d, 0xd7, 0x20, 0x78, 0x01, 0x01, 0x00,
+ 0x00, 0xff, 0xff, 0x00, 0x00, 0x00, 0x01, 0x95,
+ 0x11, 0x78, 0x01, 0x01, 0x15, 0x01, 0xea, 0xfe,
+ 0x74, 0x72, 0x65, 0x65, 0x20, 0x36, 0x65, 0x66,
+ 0x31, 0x39, 0x62, 0x34, 0x31, 0x32, 0x32, 0x35,
+ 0x63, 0x35, 0x33, 0x36, 0x39, 0x66, 0x31, 0x63,
+ 0x31, 0x30, 0x34, 0x64, 0x34, 0x35, 0x64, 0x38,
+ 0x64, 0x38, 0x35, 0x65, 0x66, 0x61, 0x39, 0x62,
+ 0x30, 0x35, 0x37, 0x62, 0x35, 0x33, 0x62, 0x31,
+ 0x34, 0x62, 0x34, 0x62, 0x39, 0x62, 0x39, 0x33,
+ 0x39, 0x64, 0x64, 0x37, 0x34, 0x64, 0x65, 0x63,
+ 0x63, 0x35, 0x33, 0x32, 0x31, 0x0a, 0x70, 0x61,
+ 0x72, 0x65, 0x6e, 0x74, 0x20, 0x37, 0x35, 0x62,
+ 0x66, 0x30, 0x63, 0x34, 0x37, 0x61, 0x65, 0x34,
+ 0x62, 0x62, 0x33, 0x30, 0x38, 0x65, 0x37, 0x63,
+ 0x63, 0x32, 0x34, 0x38, 0x32, 0x65, 0x32, 0x32,
+ 0x65, 0x66, 0x61, 0x65, 0x33, 0x37, 0x38, 0x37,
+ 0x61, 0x39, 0x36, 0x38, 0x34, 0x38, 0x62, 0x64,
+ 0x31, 0x37, 0x34, 0x39, 0x35, 0x36, 0x37, 0x31,
+ 0x34, 0x37, 0x31, 0x35, 0x32, 0x34, 0x36, 0x64,
+ 0x64, 0x62, 0x64, 0x35, 0x34, 0x0a, 0x61, 0x75,
+ 0x74, 0x68, 0x6f, 0x72, 0x20, 0x41, 0x20, 0x55,
+ 0x20, 0x54, 0x68, 0x6f, 0x72, 0x20, 0x3c, 0x61,
+ 0x75, 0x74, 0x68, 0x6f, 0x72, 0x40, 0x65, 0x78,
+ 0x61, 0x6d, 0x70, 0x6c, 0x65, 0x2e, 0x63, 0x6f,
+ 0x6d, 0x3e, 0x20, 0x31, 0x32, 0x33, 0x34, 0x35,
+ 0x36, 0x37, 0x38, 0x39, 0x30, 0x20, 0x2b, 0x30,
+ 0x30, 0x30, 0x30, 0x0a, 0x63, 0x6f, 0x6d, 0x6d,
+ 0x69, 0x74, 0x74, 0x65, 0x72, 0x20, 0x43, 0x20,
+ 0x4f, 0x20, 0x4d, 0x69, 0x74, 0x74, 0x65, 0x72,
+ 0x20, 0x3c, 0x63, 0x6f, 0x6d, 0x6d, 0x69, 0x74,
+ 0x74, 0x65, 0x72, 0x40, 0x65, 0x78, 0x61, 0x6d,
+ 0x70, 0x6c, 0x65, 0x2e, 0x63, 0x6f, 0x6d, 0x3e,
+ 0x20, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37,
+ 0x38, 0x39, 0x30, 0x20, 0x2b, 0x30, 0x30, 0x30,
+ 0x30, 0x0a, 0x0a, 0x45, 0x6d, 0x70, 0x74, 0x79,
+ 0x20, 0x74, 0x72, 0x65, 0x65, 0x20, 0x63, 0x6f,
+ 0x6d, 0x6d, 0x69, 0x74, 0x0a, 0x6d, 0x6d, 0x51,
+ 0x9a, 0xc9, 0x11, 0x76, 0x61, 0xa3, 0x89, 0x49,
+ 0xb7, 0xa1, 0x58, 0xc6, 0x1d, 0x8c, 0x33, 0x75,
+ 0x8d, 0x7e, 0x4d, 0x8e, 0x58, 0x91, 0xf8, 0x5c,
+ 0x57, 0xd9, 0x89, 0x9e, 0xb8, 0xd2, 0x9a, 0xd8,
+ 0xc9
+};
+
static const struct fast_pack fast_packs[] = {
{
.format_id = GIT_SHA1_FORMAT_ID,
@@ -253,6 +337,13 @@ static const struct fast_pack fast_packs[] = {
.suffix_len = sizeof(fast_pack_sha1_suffix),
.commit_oid = "aac43daf40d0377af31aa9c798a4ae8a31b55c1d",
},
+ {
+ .format_id = GIT_SHA256_FORMAT_ID,
+ .suffix = fast_pack_sha256_suffix,
+ .suffix_len = sizeof(fast_pack_sha256_suffix),
+ .commit_oid = "63c46ca51267b1d45be69a044bb84b4bf0559f09"
+ "d727f861d2ae94ddebdddbc9",
+ },
};
/*
--
gitgitgadget
^ permalink raw reply related
* [PATCH v2 08/11] test-tool synthesize: precompute pack for 4 GiB + 1
From: Johannes Schindelin via GitGitGadget @ 2026-05-04 17:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Torsten Bögershausen, Jeff King,
Johannes Schindelin, Johannes Schindelin
In-Reply-To: <pull.2102.v2.git.1777914508.gitgitgadget@gmail.com>
From: Johannes Schindelin <johannes.schindelin@gmx.de>
The synthesize helper hashes roughly 8 GiB of data through SHA-1 to
produce a 4 GiB + 1 pack (4 GiB for the pack checksum, 4 GiB for
the blob OID). Since the blob content is all NUL bytes, every byte
in the resulting pack file is deterministic for a given blob size and
hash algorithm.
Add a fast path that writes the pack from precomputed constants:
a 25-byte prefix (pack header, object header, zlib header, first
block header), the zero-filled bulk with periodic 5-byte deflate
block headers, and a 513-byte suffix (tree, two commits, empty tree,
pack SHA-1 checksum). This eliminates all SHA-1 and adler32
computation, making the helper purely I/O-bound.
The precomputed constants are stored in a struct fast_pack array
keyed by hash algorithm format_id, so that adding SHA-256 support
later requires only adding another array entry with its suffix.
The constants were generated by running the generic path and
extracting the non-zero bytes from the resulting pack file.
Benchmarks generating a 4 GiB + 1 pack (3 runs each, SHA1DC on
x86_64):
generic path: 88s / 81s / 140s
fast path: 14s / 13s / 15s
On CI, where t5608 currently takes 200-850 seconds depending on the
job, the fast path cuts the pack-generation phase from minutes to
seconds, leaving only the clone operations themselves.
Assisted-by: Claude Opus 4.6
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
t/helper/test-synthesize.c | 202 ++++++++++++++++++++++++++++++++++++-
1 file changed, 201 insertions(+), 1 deletion(-)
diff --git a/t/helper/test-synthesize.c b/t/helper/test-synthesize.c
index e2faaad7b4..83c40ee02a 100644
--- a/t/helper/test-synthesize.c
+++ b/t/helper/test-synthesize.c
@@ -112,6 +112,201 @@ static void write_pack_object(FILE *f, struct git_hash_ctx *pack_ctx,
algo->final_oid_fn(oid, &ctx);
}
+/*
+ * Fast path: precomputed pack data for a 4 GiB + 1 all-NUL blob.
+ *
+ * The generated pack is almost entirely zeros with a small constant
+ * prefix, periodic deflate block headers, and a constant suffix
+ * containing the tree, two commits, and the pack checksum. Because
+ * every byte is deterministic for a given blob size and hash algorithm,
+ * we can write the pack without computing any hashes at all, reducing
+ * runtime from minutes of hash computation to seconds of pure I/O.
+ *
+ * The blob is stored as an uncompressed deflate stream: a two-byte
+ * zlib header, then 65538 blocks of up to 0xffff bytes each, followed
+ * by an adler32 checksum. The pack header and deflate framing are
+ * shared across hash algorithms; only the suffix (which contains OIDs
+ * and the pack checksum) differs.
+ *
+ * Constants were generated by running the generic path and extracting
+ * the non-zero bytes from the resulting pack file.
+ */
+
+#define FAST_PACK_4G1_BLOB_SIZE ((size_t)4 * 1024 * 1024 * 1024 + 1)
+#define FAST_PACK_4G1_N_FULL_BLOCKS 65537
+
+/*
+ * Per-hash-algorithm constants for the fast path. The prefix and
+ * deflate block structure are identical across algorithms; only the
+ * suffix (tree, commits, pack checksum) and the commit OID differ.
+ */
+struct fast_pack {
+ uint32_t format_id;
+ const unsigned char *suffix;
+ size_t suffix_len;
+ const char *commit_oid;
+};
+
+/* Pack header + pack object header + zlib header + first block header */
+static const unsigned char fast_pack_prefix[] = {
+ /* PACK header: signature, version 2, 5 objects */
+ 0x50, 0x41, 0x43, 0x4b, 0x00, 0x00, 0x00, 0x02,
+ 0x00, 0x00, 0x00, 0x05,
+ /* pack object header: blob, size = 4294967297 */
+ 0xb1, 0x80, 0x80, 0x80, 0x80, 0x01,
+ /* zlib header: CMF=0x78, FLG=0x01 */
+ 0x78, 0x01,
+ /* first non-final block header: BFINAL=0, LEN=0xffff, NLEN=0x0000 */
+ 0x00, 0xff, 0xff, 0x00, 0x00
+};
+
+/* Every non-final deflate block header is identical */
+static const unsigned char fast_pack_block_header[] = {
+ 0x00, 0xff, 0xff, 0x00, 0x00
+};
+
+/* Final block (2 data bytes) + adler32 of 4294967297 NUL bytes */
+static const unsigned char fast_pack_final_block[] = {
+ /* BFINAL=1, LEN=2, NLEN=0xfffd */
+ 0x01, 0x02, 0x00, 0xfd, 0xff,
+ /* 2 NUL data bytes */
+ 0x00, 0x00,
+ /* adler32 */
+ 0x00, 0xe2, 0x00, 0x01
+};
+
+/*
+ * SHA-1 suffix: tree, commit, empty tree, final commit, pack checksum.
+ */
+static const unsigned char fast_pack_sha1_suffix[] = {
+ 0xa0, 0x02, 0x78, 0x01, 0x01, 0x20, 0x00, 0xdf,
+ 0xff, 0x31, 0x30, 0x30, 0x36, 0x34, 0x34, 0x20,
+ 0x66, 0x69, 0x6c, 0x65, 0x00, 0x3e, 0xb7, 0xfe,
+ 0xb1, 0x41, 0x3c, 0x75, 0x7f, 0x0d, 0x81, 0x81,
+ 0xde, 0xb2, 0x8d, 0x1d, 0xab, 0x03, 0xd6, 0x48,
+ 0x46, 0xb4, 0xb4, 0x0c, 0x60, 0x95, 0x0b, 0x78,
+ 0x01, 0x01, 0xb5, 0x00, 0x4a, 0xff, 0x74, 0x72,
+ 0x65, 0x65, 0x20, 0x63, 0x36, 0x38, 0x33, 0x66,
+ 0x63, 0x63, 0x37, 0x64, 0x31, 0x64, 0x38, 0x33,
+ 0x65, 0x66, 0x32, 0x66, 0x65, 0x31, 0x61, 0x66,
+ 0x35, 0x35, 0x32, 0x31, 0x35, 0x64, 0x30, 0x31,
+ 0x36, 0x38, 0x64, 0x62, 0x35, 0x32, 0x61, 0x33,
+ 0x61, 0x33, 0x62, 0x0a, 0x61, 0x75, 0x74, 0x68,
+ 0x6f, 0x72, 0x20, 0x41, 0x20, 0x55, 0x20, 0x54,
+ 0x68, 0x6f, 0x72, 0x20, 0x3c, 0x61, 0x75, 0x74,
+ 0x68, 0x6f, 0x72, 0x40, 0x65, 0x78, 0x61, 0x6d,
+ 0x70, 0x6c, 0x65, 0x2e, 0x63, 0x6f, 0x6d, 0x3e,
+ 0x20, 0x31, 0x32, 0x33, 0x34, 0x35, 0x36, 0x37,
+ 0x38, 0x39, 0x30, 0x20, 0x2b, 0x30, 0x30, 0x30,
+ 0x30, 0x0a, 0x63, 0x6f, 0x6d, 0x6d, 0x69, 0x74,
+ 0x74, 0x65, 0x72, 0x20, 0x43, 0x20, 0x4f, 0x20,
+ 0x4d, 0x69, 0x74, 0x74, 0x65, 0x72, 0x20, 0x3c,
+ 0x63, 0x6f, 0x6d, 0x6d, 0x69, 0x74, 0x74, 0x65,
+ 0x72, 0x40, 0x65, 0x78, 0x61, 0x6d, 0x70, 0x6c,
+ 0x65, 0x2e, 0x63, 0x6f, 0x6d, 0x3e, 0x20, 0x31,
+ 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, 0x38, 0x39,
+ 0x30, 0x20, 0x2b, 0x30, 0x30, 0x30, 0x30, 0x0a,
+ 0x0a, 0x4c, 0x61, 0x72, 0x67, 0x65, 0x20, 0x62,
+ 0x6c, 0x6f, 0x62, 0x20, 0x63, 0x6f, 0x6d, 0x6d,
+ 0x69, 0x74, 0x0a, 0xc6, 0x55, 0x37, 0x6b, 0x20,
+ 0x78, 0x01, 0x01, 0x00, 0x00, 0xff, 0xff, 0x00,
+ 0x00, 0x00, 0x01, 0x95, 0x0e, 0x78, 0x01, 0x01,
+ 0xe5, 0x00, 0x1a, 0xff, 0x74, 0x72, 0x65, 0x65,
+ 0x20, 0x34, 0x62, 0x38, 0x32, 0x35, 0x64, 0x63,
+ 0x36, 0x34, 0x32, 0x63, 0x62, 0x36, 0x65, 0x62,
+ 0x39, 0x61, 0x30, 0x36, 0x30, 0x65, 0x35, 0x34,
+ 0x62, 0x66, 0x38, 0x64, 0x36, 0x39, 0x32, 0x38,
+ 0x38, 0x66, 0x62, 0x65, 0x65, 0x34, 0x39, 0x30,
+ 0x34, 0x0a, 0x70, 0x61, 0x72, 0x65, 0x6e, 0x74,
+ 0x20, 0x63, 0x35, 0x62, 0x32, 0x31, 0x63, 0x36,
+ 0x31, 0x31, 0x61, 0x61, 0x35, 0x39, 0x34, 0x65,
+ 0x63, 0x39, 0x66, 0x64, 0x37, 0x65, 0x39, 0x32,
+ 0x63, 0x66, 0x39, 0x36, 0x34, 0x38, 0x39, 0x31,
+ 0x34, 0x63, 0x61, 0x34, 0x63, 0x32, 0x34, 0x31,
+ 0x32, 0x0a, 0x61, 0x75, 0x74, 0x68, 0x6f, 0x72,
+ 0x20, 0x41, 0x20, 0x55, 0x20, 0x54, 0x68, 0x6f,
+ 0x72, 0x20, 0x3c, 0x61, 0x75, 0x74, 0x68, 0x6f,
+ 0x72, 0x40, 0x65, 0x78, 0x61, 0x6d, 0x70, 0x6c,
+ 0x65, 0x2e, 0x63, 0x6f, 0x6d, 0x3e, 0x20, 0x31,
+ 0x32, 0x33, 0x34, 0x35, 0x36, 0x37, 0x38, 0x39,
+ 0x30, 0x20, 0x2b, 0x30, 0x30, 0x30, 0x30, 0x0a,
+ 0x63, 0x6f, 0x6d, 0x6d, 0x69, 0x74, 0x74, 0x65,
+ 0x72, 0x20, 0x43, 0x20, 0x4f, 0x20, 0x4d, 0x69,
+ 0x74, 0x74, 0x65, 0x72, 0x20, 0x3c, 0x63, 0x6f,
+ 0x6d, 0x6d, 0x69, 0x74, 0x74, 0x65, 0x72, 0x40,
+ 0x65, 0x78, 0x61, 0x6d, 0x70, 0x6c, 0x65, 0x2e,
+ 0x63, 0x6f, 0x6d, 0x3e, 0x20, 0x31, 0x32, 0x33,
+ 0x34, 0x35, 0x36, 0x37, 0x38, 0x39, 0x30, 0x20,
+ 0x2b, 0x30, 0x30, 0x30, 0x30, 0x0a, 0x0a, 0x45,
+ 0x6d, 0x70, 0x74, 0x79, 0x20, 0x74, 0x72, 0x65,
+ 0x65, 0x20, 0x63, 0x6f, 0x6d, 0x6d, 0x69, 0x74,
+ 0x0a, 0xaa, 0xb8, 0x45, 0x01, 0x8e, 0xfc, 0xf0,
+ 0x2f, 0x9c, 0xc5, 0xcc, 0x4f, 0x6a, 0x1a, 0xc9,
+ 0x2b, 0x23, 0xa9, 0xff, 0x91, 0x06, 0xc2, 0x70,
+ 0xe3
+};
+
+static const struct fast_pack fast_packs[] = {
+ {
+ .format_id = GIT_SHA1_FORMAT_ID,
+ .suffix = fast_pack_sha1_suffix,
+ .suffix_len = sizeof(fast_pack_sha1_suffix),
+ .commit_oid = "aac43daf40d0377af31aa9c798a4ae8a31b55c1d",
+ },
+};
+
+/*
+ * Try the fast path for known blob sizes. Returns 1 if the pack was
+ * written from precomputed constants, 0 if the caller should fall
+ * through to the generic path.
+ */
+static int generate_fast_pack(const char *path, size_t blob_size,
+ const struct git_hash_algo *algo)
+{
+ const struct fast_pack *fp = NULL;
+ FILE *f;
+ size_t i;
+
+ if (blob_size != FAST_PACK_4G1_BLOB_SIZE)
+ return 0;
+
+ for (i = 0; i < ARRAY_SIZE(fast_packs); i++) {
+ if (fast_packs[i].format_id == algo->format_id) {
+ fp = &fast_packs[i];
+ break;
+ }
+ }
+ if (!fp)
+ return 0;
+
+ f = xfopen(path, "wb");
+
+ fwrite_or_die(f, fast_pack_prefix, sizeof(fast_pack_prefix));
+
+ /* First full block: 0xffff zero bytes (header already in prefix) */
+ fwrite_or_die(f, zeros, BLOCK_SIZE);
+
+ /* Remaining non-final full blocks */
+ for (i = 1; i < FAST_PACK_4G1_N_FULL_BLOCKS; i++) {
+ fwrite_or_die(f, fast_pack_block_header,
+ sizeof(fast_pack_block_header));
+ fwrite_or_die(f, zeros, BLOCK_SIZE);
+ }
+
+ /* Final block (2 data bytes) + adler32 */
+ fwrite_or_die(f, fast_pack_final_block,
+ sizeof(fast_pack_final_block));
+
+ /* Tree, commits, and pack checksum */
+ fwrite_or_die(f, fp->suffix, fp->suffix_len);
+
+ if (fclose(f))
+ die_errno(_("could not close '%s'"), path);
+
+ printf("%s\n", fp->commit_oid);
+ return 1;
+}
+
/*
* Generate a pack file with a single large (>4GB) reachable object.
*
@@ -127,7 +322,7 @@ static void write_pack_object(FILE *f, struct git_hash_ctx *pack_ctx,
static int generate_pack_with_large_object(const char *path, size_t blob_size,
const struct git_hash_algo *algo)
{
- FILE *f = xfopen(path, "wb");
+ FILE *f;
struct git_hash_ctx pack_ctx;
unsigned char pack_hash[GIT_MAX_RAWSZ];
struct object_id blob_oid, tree_oid, commit_oid, empty_tree_oid, final_commit_oid;
@@ -139,6 +334,11 @@ static int generate_pack_with_large_object(const char *path, size_t blob_size,
.hdr_entries = htonl(object_count),
};
+ if (generate_fast_pack(path, blob_size, algo))
+ return 0;
+
+ f = xfopen(path, "wb");
+
algo->init_fn(&pack_ctx);
/* Write pack header */
--
gitgitgadget
^ permalink raw reply related
* [PATCH v2 07/11] test-tool synthesize: use the unsafe hash for speed
From: Johannes Schindelin via GitGitGadget @ 2026-05-04 17:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Torsten Bögershausen, Jeff King,
Johannes Schindelin, Johannes Schindelin
In-Reply-To: <pull.2102.v2.git.1777914508.gitgitgadget@gmail.com>
From: Johannes Schindelin <johannes.schindelin@gmx.de>
Jeff King pointed out on the mailing list [1] that t5608's new >4GB
test cases dominate the entire test suite runtime: 160 seconds on his
laptop when the rest of the suite finishes in under 90 seconds, and
305-850 seconds across CI jobs. The bottleneck is that the synthesize
helper hashes roughly 8 GB of data through SHA-1 (4 GB for the pack
checksum plus 4 GB for the blob OID) for a 4 GB+1 blob.
Since the helper generates known test data, collision detection is
unnecessary. Switch from repo->hash_algo to unsafe_hash_algo(), which
uses hardware-accelerated SHA-1 (via OpenSSL or Apple CommonCrypto)
when available.
Benchmarks on an x86_64 machine generating a 4 GB+1 pack (2 runs
each, interleaved):
SHA-1 backend Run 1 Run 2
SHA1DC (safe) 75s 80s
OpenSSL (unsafe) 21s 19s
The effect scales linearly. At 64 MB with 10 randomized interleaved
runs, the OpenSSL unsafe backend shows a 5.4x improvement (median
0.202s vs 1.088s) with tight variance (stdev 0.028s vs 0.095s).
The speedup is only realized when the build has a fast unsafe backend
compiled in. The CI's linux-TEST-vars job already sets
OPENSSL_SHA1_UNSAFE=YesPlease; macOS benefits from Apple CommonCrypto
when configured. On builds without a separate unsafe backend (such as
the default Windows builds), unsafe_hash_algo() returns the regular
collision-detecting implementation and the change is a no-op.
[1] https://lore.kernel.org/git/20260501063805.GA2038915@coredump.intra.peff.net/
Assisted-by: Claude Opus 4.6
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
t/helper/test-synthesize.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/t/helper/test-synthesize.c b/t/helper/test-synthesize.c
index 3ce7078078..e2faaad7b4 100644
--- a/t/helper/test-synthesize.c
+++ b/t/helper/test-synthesize.c
@@ -217,7 +217,7 @@ static int cmd__synthesize__pack(int argc, const char **argv,
setup_git_directory_gently(&non_git);
repo = the_repository;
- algo = repo->hash_algo;
+ algo = unsafe_hash_algo(repo->hash_algo);
argc = parse_options(argc, argv, NULL, options, usage,
PARSE_OPT_KEEP_ARGV0);
--
gitgitgadget
^ permalink raw reply related
* [PATCH v2 06/11] t5608: add regression test for >4GB object clone
From: Johannes Schindelin via GitGitGadget @ 2026-05-04 17:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Torsten Bögershausen, Jeff King,
Johannes Schindelin, Johannes Schindelin
In-Reply-To: <pull.2102.v2.git.1777914508.gitgitgadget@gmail.com>
From: Johannes Schindelin <johannes.schindelin@gmx.de>
The shift overflow bug in index-pack and unpack-objects caused incorrect
object size calculation when the encoded size required more than 32 bits
of shift. This would result in corrupted or failed unpacking of objects
larger than 4GB.
Add a test that creates a pack file containing a 4GB+ blob using the
new 'test-tool synthesize pack --reachable-large' command, then clones
the repository to verify the fix works correctly.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
t/t5608-clone-2gb.sh | 37 +++++++++++++++++++++++++++++++++++++
1 file changed, 37 insertions(+)
diff --git a/t/t5608-clone-2gb.sh b/t/t5608-clone-2gb.sh
index 87a8cd9f98..af93302dde 100755
--- a/t/t5608-clone-2gb.sh
+++ b/t/t5608-clone-2gb.sh
@@ -49,4 +49,41 @@ test_expect_success 'clone - with worktree, file:// protocol' '
'
+test_expect_success SIZE_T_IS_64BIT 'set up repo with >4GB object' '
+ large_blob_size=$((4*1024*1024*1024+1)) &&
+ git init --bare 4gb-repo &&
+ head_oid=$(test-tool synthesize pack \
+ --reachable-large "$large_blob_size" \
+ 4gb-repo/objects/pack/test.pack) &&
+ git -C 4gb-repo index-pack objects/pack/test.pack &&
+ git -C 4gb-repo update-ref refs/heads/main $head_oid &&
+ git -C 4gb-repo symbolic-ref HEAD refs/heads/main
+'
+
+test_expect_success SIZE_T_IS_64BIT 'clone >4GB object via unpack-objects' '
+ # The synthesized pack has five objects, so a large unpack limit keeps
+ # fetch-pack on the unpack-objects path.
+ git -c fetch.unpackLimit=100 clone --bare \
+ "file://$(pwd)/4gb-repo" 4gb-clone-unpack &&
+
+ # Verify the large blob survived the clone by comparing its OID
+ # between source and clone. We cannot use "cat-file -s" because
+ # object_info.sizep is still unsigned long, which truncates >4GB
+ # sizes on Windows. OID equality proves content integrity since
+ # the clone already verified checksums via index-pack/unpack-objects.
+ source_blob=$(git -C 4gb-repo rev-parse main^:file) &&
+ clone_blob=$(git -C 4gb-clone-unpack rev-parse main^:file) &&
+ test "$source_blob" = "$clone_blob"
+'
+
+test_expect_success SIZE_T_IS_64BIT 'clone with >4GB object via index-pack' '
+ # Force fetch-pack to hand the pack to index-pack instead.
+ git -c fetch.unpackLimit=1 clone --bare \
+ "file://$(pwd)/4gb-repo" 4gb-clone-index &&
+
+ source_blob=$(git -C 4gb-repo rev-parse main^:file) &&
+ clone_blob=$(git -C 4gb-clone-index rev-parse main^:file) &&
+ test "$source_blob" = "$clone_blob"
+'
+
test_done
--
gitgitgadget
^ permalink raw reply related
* [PATCH v2 05/11] test-tool: add a helper to synthesize large packfiles
From: Johannes Schindelin via GitGitGadget @ 2026-05-04 17:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Torsten Bögershausen, Jeff King,
Johannes Schindelin, Johannes Schindelin
In-Reply-To: <pull.2102.v2.git.1777914508.gitgitgadget@gmail.com>
From: Johannes Schindelin <johannes.schindelin@gmx.de>
To test Git's behavior with very large pack files, we need a way to
generate such files quickly.
A naive approach using only readily-available Git commands would take
over 10 hours for a 4GB pack file, which is prohibitive.
Side-stepping Git's machinery and actual zlib compression by writing
uncompressed content with the appropriate zlib header makes things
much faster. The fastest method using this approach generates many
small, unreachable blob objects and takes about 1.5 minutes for 4GB.
However, this cannot be used because we need to test git clone, which
requires a reachable commit history.
Generating many reachable commits with small, uncompressed blobs takes
about 4 minutes for 4GB. But this approach 1) does not reproduce the
issues we want to fix (which require individual objects larger than
4GB) and 2) is comparatively slow because of the many SHA-1
calculations.
The approach taken here generates a single large blob (filled with NUL
bytes), along with the trees and commits needed to make it reachable.
This takes about 2.5 minutes for 4.5GB, which is the fastest option
that produces a valid, clonable repository with an object large enough
to trigger the bugs we want to test.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
Makefile | 1 +
compat/zlib-compat.h | 2 +
t/helper/meson.build | 1 +
t/helper/test-synthesize.c | 250 +++++++++++++++++++++++++++++++++++++
t/helper/test-tool.c | 1 +
t/helper/test-tool.h | 1 +
6 files changed, 256 insertions(+)
create mode 100644 t/helper/test-synthesize.c
diff --git a/Makefile b/Makefile
index cedc234173..85405cb5b8 100644
--- a/Makefile
+++ b/Makefile
@@ -872,6 +872,7 @@ TEST_BUILTINS_OBJS += test-submodule-config.o
TEST_BUILTINS_OBJS += test-submodule-nested-repo-config.o
TEST_BUILTINS_OBJS += test-submodule.o
TEST_BUILTINS_OBJS += test-subprocess.o
+TEST_BUILTINS_OBJS += test-synthesize.o
TEST_BUILTINS_OBJS += test-trace2.o
TEST_BUILTINS_OBJS += test-truncate.o
TEST_BUILTINS_OBJS += test-userdiff.o
diff --git a/compat/zlib-compat.h b/compat/zlib-compat.h
index ac08276622..5078c5ef6c 100644
--- a/compat/zlib-compat.h
+++ b/compat/zlib-compat.h
@@ -7,6 +7,8 @@
# define z_stream_s zng_stream_s
# define gz_header_s zng_gz_header_s
+# define adler32(adler, buf, len) zng_adler32(adler, buf, len)
+
# define crc32(crc, buf, len) zng_crc32(crc, buf, len)
# define inflate(strm, bits) zng_inflate(strm, bits)
diff --git a/t/helper/meson.build b/t/helper/meson.build
index 675e64c010..3235f10ab8 100644
--- a/t/helper/meson.build
+++ b/t/helper/meson.build
@@ -69,6 +69,7 @@ test_tool_sources = [
'test-submodule-nested-repo-config.c',
'test-submodule.c',
'test-subprocess.c',
+ 'test-synthesize.c',
'test-tool.c',
'test-trace2.c',
'test-truncate.c',
diff --git a/t/helper/test-synthesize.c b/t/helper/test-synthesize.c
new file mode 100644
index 0000000000..3ce7078078
--- /dev/null
+++ b/t/helper/test-synthesize.c
@@ -0,0 +1,250 @@
+#define USE_THE_REPOSITORY_VARIABLE
+
+#include "test-tool.h"
+#include "git-compat-util.h"
+#include "git-zlib.h"
+#include "hash.h"
+#include "hex.h"
+#include "object-file.h"
+#include "object.h"
+#include "pack.h"
+#include "parse-options.h"
+#include "parse.h"
+#include "repository.h"
+#include "setup.h"
+#include "strbuf.h"
+#include "write-or-die.h"
+
+#define BLOCK_SIZE 0xffff
+static const unsigned char zeros[BLOCK_SIZE];
+
+/*
+ * Write data as an uncompressed zlib stream.
+ * For data larger than 64KB, writes multiple uncompressed blocks.
+ * If data is NULL, writes zeros.
+ * Updates the pack checksum context.
+ */
+static void write_uncompressed_zlib(FILE *f, struct git_hash_ctx *pack_ctx,
+ const void *data, size_t len,
+ const struct git_hash_algo *algo)
+{
+ unsigned char zlib_header[2] = { 0x78, 0x01 }; /* CMF, FLG */
+ unsigned char block_header[5];
+ const unsigned char *p = data;
+ size_t remaining = len;
+ uint32_t adler = 1L; /* adler32 initial value */
+ unsigned char adler_buf[4];
+
+ /* Write zlib header */
+ fwrite_or_die(f, zlib_header, sizeof(zlib_header));
+ algo->update_fn(pack_ctx, zlib_header, 2);
+
+ /* Write uncompressed blocks (max 64KB each) */
+ do {
+ size_t block_len = remaining > BLOCK_SIZE ? BLOCK_SIZE : remaining;
+ int is_final = (block_len == remaining);
+ const unsigned char *block_data = data ? p : zeros;
+
+ block_header[0] = is_final ? 0x01 : 0x00;
+ block_header[1] = block_len & 0xff;
+ block_header[2] = (block_len >> 8) & 0xff;
+ block_header[3] = block_header[1] ^ 0xff;
+ block_header[4] = block_header[2] ^ 0xff;
+
+ fwrite_or_die(f, block_header, sizeof(block_header));
+ algo->update_fn(pack_ctx, block_header, 5);
+
+ if (block_len) {
+ fwrite_or_die(f, block_data, block_len);
+ algo->update_fn(pack_ctx, block_data, block_len);
+ adler = adler32(adler, block_data, block_len);
+ }
+
+ if (data)
+ p += block_len;
+ remaining -= block_len;
+ } while (remaining > 0);
+
+ /* Write adler32 checksum */
+ put_be32(adler_buf, adler);
+ fwrite_or_die(f, adler_buf, sizeof(adler_buf));
+ algo->update_fn(pack_ctx, adler_buf, 4);
+}
+
+/*
+ * Write an uncompressed object to the pack file.
+ * If `data == NULL`, it is treated like a buffer to NUL bytes.
+ * Updates the pack checksum context.
+ */
+static void write_pack_object(FILE *f, struct git_hash_ctx *pack_ctx,
+ enum object_type type,
+ const void *data, size_t len,
+ struct object_id *oid,
+ const struct git_hash_algo *algo)
+{
+ unsigned char pack_header[MAX_PACK_OBJECT_HEADER];
+ char object_header[32];
+ int pack_header_len, object_header_len;
+ struct git_hash_ctx ctx;
+
+ /* Write pack object header */
+ pack_header_len = encode_in_pack_object_header(pack_header,
+ sizeof(pack_header),
+ type, len);
+ fwrite_or_die(f, pack_header, pack_header_len);
+ algo->update_fn(pack_ctx, pack_header, pack_header_len);
+
+ /* Write the data as uncompressed zlib */
+ write_uncompressed_zlib(f, pack_ctx, data, len, algo);
+
+ algo->init_fn(&ctx);
+ object_header_len = format_object_header(object_header,
+ sizeof(object_header),
+ type, len);
+ algo->update_fn(&ctx, object_header, object_header_len);
+ if (data)
+ algo->update_fn(&ctx, data, len);
+ else {
+ for (size_t i = len / BLOCK_SIZE; i; i--)
+ algo->update_fn(&ctx, zeros, BLOCK_SIZE);
+ algo->update_fn(&ctx, zeros, len % BLOCK_SIZE);
+ }
+ algo->final_oid_fn(oid, &ctx);
+}
+
+/*
+ * Generate a pack file with a single large (>4GB) reachable object.
+ *
+ * Creates:
+ * 1. A large blob (all NUL bytes)
+ * 2. A tree containing that blob as "file"
+ * 3. A commit using that tree
+ * 4. The empty tree
+ * 5. A child commit using the empty tree
+ *
+ * This is useful for testing that Git can handle objects larger than 4GB.
+ */
+static int generate_pack_with_large_object(const char *path, size_t blob_size,
+ const struct git_hash_algo *algo)
+{
+ FILE *f = xfopen(path, "wb");
+ struct git_hash_ctx pack_ctx;
+ unsigned char pack_hash[GIT_MAX_RAWSZ];
+ struct object_id blob_oid, tree_oid, commit_oid, empty_tree_oid, final_commit_oid;
+ struct strbuf buf = STRBUF_INIT;
+ const uint32_t object_count = 5;
+ struct pack_header pack_header = {
+ .hdr_signature = htonl(PACK_SIGNATURE),
+ .hdr_version = htonl(PACK_VERSION),
+ .hdr_entries = htonl(object_count),
+ };
+
+ algo->init_fn(&pack_ctx);
+
+ /* Write pack header */
+ fwrite_or_die(f, &pack_header, sizeof(pack_header));
+ algo->update_fn(&pack_ctx, &pack_header, sizeof(pack_header));
+
+ /* 1. Write the large blob */
+ write_pack_object(f, &pack_ctx, OBJ_BLOB, NULL, blob_size, &blob_oid, algo);
+
+ /* 2. Write tree containing the blob as "file" */
+ strbuf_addf(&buf, "100644 file%c", '\0');
+ strbuf_add(&buf, blob_oid.hash, algo->rawsz);
+ write_pack_object(f, &pack_ctx, OBJ_TREE, buf.buf, buf.len, &tree_oid, algo);
+
+ /* 3. Write commit using that tree */
+ strbuf_reset(&buf);
+ strbuf_addf(&buf,
+ "tree %s\n"
+ "author A U Thor <author@example.com> 1234567890 +0000\n"
+ "committer C O Mitter <committer@example.com> 1234567890 +0000\n"
+ "\n"
+ "Large blob commit\n",
+ oid_to_hex(&tree_oid));
+ write_pack_object(f, &pack_ctx, OBJ_COMMIT, buf.buf, buf.len, &commit_oid, algo);
+
+ /* 4. Write the empty tree */
+ write_pack_object(f, &pack_ctx, OBJ_TREE, "", 0, &empty_tree_oid, algo);
+
+ /* 5. Write final commit using empty tree, with previous commit as parent */
+ strbuf_reset(&buf);
+ strbuf_addf(&buf,
+ "tree %s\n"
+ "parent %s\n"
+ "author A U Thor <author@example.com> 1234567890 +0000\n"
+ "committer C O Mitter <committer@example.com> 1234567890 +0000\n"
+ "\n"
+ "Empty tree commit\n",
+ oid_to_hex(&empty_tree_oid),
+ oid_to_hex(&commit_oid));
+ write_pack_object(f, &pack_ctx, OBJ_COMMIT, buf.buf, buf.len, &final_commit_oid, algo);
+
+ /* Write pack trailer (checksum) */
+ algo->final_fn(pack_hash, &pack_ctx);
+ fwrite_or_die(f, pack_hash, algo->rawsz);
+ if (fclose(f))
+ die_errno(_("could not close '%s'"), path);
+
+ strbuf_release(&buf);
+
+ /* Print the final commit OID so caller can set up refs */
+ printf("%s\n", oid_to_hex(&final_commit_oid));
+
+ return 0;
+}
+
+static int cmd__synthesize__pack(int argc, const char **argv,
+ const char *prefix UNUSED,
+ struct repository *repo)
+{
+ int non_git;
+ int reachable_large = 0;
+ const struct git_hash_algo *algo;
+ size_t blob_size;
+ uintmax_t blob_size_u;
+ const char *path;
+ const char * const usage[] = {
+ "test-tool synthesize pack "
+ "--reachable-large <blob-size> <filename>",
+ NULL
+ };
+ struct option options[] = {
+ OPT_BOOL(0, "reachable-large", &reachable_large,
+ N_("write a pack with a single reachable large blob")),
+ OPT_END()
+ };
+
+ setup_git_directory_gently(&non_git);
+ repo = the_repository;
+ algo = repo->hash_algo;
+
+ argc = parse_options(argc, argv, NULL, options, usage,
+ PARSE_OPT_KEEP_ARGV0);
+ if (argc != 3 || !reachable_large)
+ usage_with_options(usage, options);
+
+ if (!git_parse_unsigned(argv[1], &blob_size_u,
+ maximum_unsigned_value_of_type(size_t)))
+ die(_("'%s' is not a valid blob size"), argv[1]);
+ blob_size = blob_size_u;
+ path = argv[2];
+
+ return !!generate_pack_with_large_object(path, blob_size, algo);
+}
+
+int cmd__synthesize(int argc, const char **argv)
+{
+ const char *prefix = NULL;
+ char const * const synthesize_usage[] = {
+ "test-tool synthesize pack <options>",
+ NULL,
+ };
+ parse_opt_subcommand_fn *fn = NULL;
+ struct option options[] = {
+ OPT_SUBCOMMAND("pack", &fn, cmd__synthesize__pack),
+ OPT_END()
+ };
+ argc = parse_options(argc, argv, prefix, options, synthesize_usage, 0);
+ return !!fn(argc, argv, prefix, NULL);
+}
diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c
index a7abc618b3..b71a22b43b 100644
--- a/t/helper/test-tool.c
+++ b/t/helper/test-tool.c
@@ -82,6 +82,7 @@ static struct test_cmd cmds[] = {
{ "submodule-config", cmd__submodule_config },
{ "submodule-nested-repo-config", cmd__submodule_nested_repo_config },
{ "subprocess", cmd__subprocess },
+ { "synthesize", cmd__synthesize },
{ "trace2", cmd__trace2 },
{ "truncate", cmd__truncate },
{ "userdiff", cmd__userdiff },
diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h
index 7f150fa1eb..f2885b33d5 100644
--- a/t/helper/test-tool.h
+++ b/t/helper/test-tool.h
@@ -75,6 +75,7 @@ int cmd__submodule(int argc, const char **argv);
int cmd__submodule_config(int argc, const char **argv);
int cmd__submodule_nested_repo_config(int argc, const char **argv);
int cmd__subprocess(int argc, const char **argv);
+int cmd__synthesize(int argc, const char **argv);
int cmd__trace2(int argc, const char **argv);
int cmd__truncate(int argc, const char **argv);
int cmd__userdiff(int argc, const char **argv);
--
gitgitgadget
^ permalink raw reply related
* [PATCH v2 04/11] delta, packfile: use size_t for delta header sizes
From: Johannes Schindelin via GitGitGadget @ 2026-05-04 17:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Torsten Bögershausen, Jeff King,
Johannes Schindelin, Johannes Schindelin
In-Reply-To: <pull.2102.v2.git.1777914508.gitgitgadget@gmail.com>
From: Johannes Schindelin <johannes.schindelin@gmx.de>
The delta header decoding functions return unsigned long, which
truncates on Windows for objects larger than 4GB. Introduce size_t
variants get_delta_hdr_size_sz() and get_size_from_delta_sz() that
preserve the full 64-bit size, and use them in packed_object_info()
where the size is needed for streaming decisions.
This was originally authored by LordKiRon <https://github.com/LordKiRon>,
who preferred not to reveal their real name and therefore agreed that I
take over authorship.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
delta.h | 14 ++++++++++++--
packfile.c | 33 ++++++++++++++++++++++++---------
2 files changed, 36 insertions(+), 11 deletions(-)
diff --git a/delta.h b/delta.h
index 8a56ec0799..fad68cfc45 100644
--- a/delta.h
+++ b/delta.h
@@ -86,8 +86,11 @@ void *patch_delta(const void *src_buf, unsigned long src_size,
* This must be called twice on the delta data buffer, first to get the
* expected source buffer size, and again to get the target buffer size.
*/
-static inline unsigned long get_delta_hdr_size(const unsigned char **datap,
- const unsigned char *top)
+/*
+ * Size_t variant that doesn't truncate - use for >4GB objects on Windows.
+ */
+static inline size_t get_delta_hdr_size_sz(const unsigned char **datap,
+ const unsigned char *top)
{
const unsigned char *data = *datap;
size_t cmd, size = 0;
@@ -98,6 +101,13 @@ static inline unsigned long get_delta_hdr_size(const unsigned char **datap,
i += 7;
} while (cmd & 0x80 && data < top);
*datap = data;
+ return size;
+}
+
+static inline unsigned long get_delta_hdr_size(const unsigned char **datap,
+ const unsigned char *top)
+{
+ size_t size = get_delta_hdr_size_sz(datap, top);
return cast_size_t_to_ulong(size);
}
diff --git a/packfile.c b/packfile.c
index fdae91dd11..4208f53046 100644
--- a/packfile.c
+++ b/packfile.c
@@ -1161,9 +1161,12 @@ unsigned long unpack_object_header_buffer(const unsigned char *buf,
return used;
}
-unsigned long get_size_from_delta(struct packed_git *p,
- struct pack_window **w_curs,
- off_t curpos)
+/*
+ * Size_t variant for >4GB delta results on Windows.
+ */
+static size_t get_size_from_delta_sz(struct packed_git *p,
+ struct pack_window **w_curs,
+ off_t curpos)
{
const unsigned char *data;
unsigned char delta_head[20], *in;
@@ -1210,10 +1213,18 @@ unsigned long get_size_from_delta(struct packed_git *p,
data = delta_head;
/* ignore base size */
- get_delta_hdr_size(&data, delta_head+sizeof(delta_head));
+ get_delta_hdr_size_sz(&data, delta_head+sizeof(delta_head));
/* Read the result size */
- return get_delta_hdr_size(&data, delta_head+sizeof(delta_head));
+ return get_delta_hdr_size_sz(&data, delta_head+sizeof(delta_head));
+}
+
+unsigned long get_size_from_delta(struct packed_git *p,
+ struct pack_window **w_curs,
+ off_t curpos)
+{
+ size_t size = get_size_from_delta_sz(p, w_curs, curpos);
+ return cast_size_t_to_ulong(size);
}
int unpack_object_header(struct packed_git *p,
@@ -1618,14 +1629,18 @@ static int packed_object_info_with_index_pos(struct packed_git *p, off_t obj_off
ret = -1;
goto out;
}
- *oi->sizep = get_size_from_delta(p, &w_curs, tmp_pos);
- if (*oi->sizep == 0) {
+ /*
+ * Use size_t variant to avoid die() on >4GB deltas.
+ * oi->sizep is unsigned long, so truncation may occur,
+ * but streaming code uses its own size_t tracking.
+ */
+ size = get_size_from_delta_sz(p, &w_curs, tmp_pos);
+ if (size == 0) {
ret = -1;
goto out;
}
- } else {
- *oi->sizep = size;
}
+ *oi->sizep = (unsigned long)size;
}
if (oi->disk_sizep || (oi->mtimep && p->is_cruft)) {
--
gitgitgadget
^ permalink raw reply related
* [PATCH v2 03/11] odb, packfile: use size_t for streaming object sizes
From: Johannes Schindelin via GitGitGadget @ 2026-05-04 17:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Torsten Bögershausen, Jeff King,
Johannes Schindelin, Johannes Schindelin
In-Reply-To: <pull.2102.v2.git.1777914508.gitgitgadget@gmail.com>
From: Johannes Schindelin <johannes.schindelin@gmx.de>
The odb_read_stream structure uses unsigned long for the size field,
which is 32-bit on Windows even in 64-bit builds. When streaming
objects larger than 4GB, the size would be truncated to zero or an
incorrect value, resulting in empty files being written to disk.
Change the size field in odb_read_stream to size_t and introduce
unpack_object_header_sz() to return sizes via size_t pointer. Since
object_info.sizep remains unsigned long for API compatibility, use
temporary variables where the types differ, with comments noting the
truncation limitation for code paths that still use unsigned long.
This was originally authored by LordKiRon <https://github.com/LordKiRon>,
who preferred not to reveal their real name and therefore agreed that I
take over authorship.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
builtin/pack-objects.c | 23 ++++++++++++++++-------
object-file.c | 10 +++++++++-
odb/streaming.c | 13 ++++++++++++-
odb/streaming.h | 2 +-
| 2 +-
pack-bitmap.c | 2 +-
pack-check.c | 6 ++++--
packfile.c | 24 +++++++++++++++---------
packfile.h | 4 ++--
9 files changed, 61 insertions(+), 25 deletions(-)
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index dd2480a73d..aa4b1cb9b8 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -629,14 +629,21 @@ static off_t write_reuse_object(struct hashfile *f, struct object_entry *entry,
struct packed_git *p = IN_PACK(entry);
struct pack_window *w_curs = NULL;
uint32_t pos;
- off_t offset;
+ off_t offset, cur;
enum object_type type = oe_type(entry);
+ enum object_type in_pack_type;
off_t datalen;
unsigned char header[MAX_PACK_OBJECT_HEADER],
dheader[MAX_PACK_OBJECT_HEADER];
unsigned hdrlen;
const unsigned hashsz = the_hash_algo->rawsz;
- unsigned long entry_size = SIZE(entry);
+ size_t entry_size;
+
+ cur = entry->in_pack_offset;
+ in_pack_type = unpack_object_header(p, &w_curs, &cur, &entry_size);
+ if (in_pack_type < 0)
+ die(_("write_reuse_object: unable to parse object header of %s"),
+ oid_to_hex(&entry->idx.oid));
if (DELTA(entry))
type = (allow_ofs_delta && DELTA(entry)->idx.offset) ?
@@ -1087,7 +1094,7 @@ static void write_reused_pack_one(struct packed_git *reuse_packfile,
{
off_t offset, next, cur;
enum object_type type;
- unsigned long size;
+ size_t size;
offset = pack_pos_to_offset(reuse_packfile, pos);
next = pack_pos_to_offset(reuse_packfile, pos + 1);
@@ -2243,7 +2250,7 @@ static void check_object(struct object_entry *entry, uint32_t object_index)
off_t ofs;
unsigned char *buf, c;
enum object_type type;
- unsigned long in_pack_size;
+ size_t in_pack_size;
buf = use_pack(p, &w_curs, entry->in_pack_offset, &avail);
@@ -2734,16 +2741,18 @@ unsigned long oe_get_size_slow(struct packing_data *pack,
struct pack_window *w_curs;
unsigned char *buf;
enum object_type type;
- unsigned long used, avail, size;
+ unsigned long used, avail;
+ size_t size;
if (e->type_ != OBJ_OFS_DELTA && e->type_ != OBJ_REF_DELTA) {
+ unsigned long sz;
packing_data_lock(&to_pack);
if (odb_read_object_info(the_repository->objects,
- &e->idx.oid, &size) < 0)
+ &e->idx.oid, &sz) < 0)
die(_("unable to get size of %s"),
oid_to_hex(&e->idx.oid));
packing_data_unlock(&to_pack);
- return size;
+ return sz;
}
p = oe_in_pack(pack, e);
diff --git a/object-file.c b/object-file.c
index 086b2b65ff..0be2981c7a 100644
--- a/object-file.c
+++ b/object-file.c
@@ -2326,6 +2326,7 @@ int odb_source_loose_read_object_stream(struct odb_read_stream **out,
struct object_info oi = OBJECT_INFO_INIT;
struct odb_loose_read_stream *st;
unsigned long mapsize;
+ unsigned long size_ul;
void *mapped;
mapped = odb_source_loose_map_object(source, oid, &mapsize);
@@ -2349,11 +2350,18 @@ int odb_source_loose_read_object_stream(struct odb_read_stream **out,
goto error;
}
- oi.sizep = &st->base.size;
+ /*
+ * object_info.sizep is unsigned long* (32-bit on Windows), but
+ * st->base.size is size_t (64-bit). Use temporary variable.
+ * Note: loose objects >4GB would still truncate here, but such
+ * large loose objects are uncommon (they'd normally be packed).
+ */
+ oi.sizep = &size_ul;
oi.typep = &st->base.type;
if (parse_loose_header(st->hdr, &oi) < 0 || st->base.type < 0)
goto error;
+ st->base.size = size_ul;
st->mapped = mapped;
st->mapsize = mapsize;
diff --git a/odb/streaming.c b/odb/streaming.c
index 5927a12954..af2adf5ce7 100644
--- a/odb/streaming.c
+++ b/odb/streaming.c
@@ -157,15 +157,26 @@ static int open_istream_incore(struct odb_read_stream **out,
.base.read = read_istream_incore,
};
struct odb_incore_read_stream *st;
+ unsigned long size_ul;
int ret;
oi.typep = &stream.base.type;
- oi.sizep = &stream.base.size;
+ /*
+ * object_info.sizep is unsigned long* (32-bit on Windows), but
+ * stream.base.size is size_t (64-bit). We use a temporary variable
+ * because the types are incompatible. Note: this path still truncates
+ * for >4GB objects, but large objects should use pack streaming
+ * (packfile_store_read_object_stream) which handles size_t properly.
+ * This incore fallback is only used for small objects or when pack
+ * streaming is unavailable.
+ */
+ oi.sizep = &size_ul;
oi.contentp = (void **)&stream.buf;
ret = odb_read_object_info_extended(odb, oid, &oi,
OBJECT_INFO_DIE_IF_CORRUPT);
if (ret)
return ret;
+ stream.base.size = size_ul;
CALLOC_ARRAY(st, 1);
*st = stream;
diff --git a/odb/streaming.h b/odb/streaming.h
index c7861f7e13..517e2ea2d3 100644
--- a/odb/streaming.h
+++ b/odb/streaming.h
@@ -21,7 +21,7 @@ struct odb_read_stream {
odb_read_stream_close_fn close;
odb_read_stream_read_fn read;
enum object_type type;
- unsigned long size; /* inflated size of full object */
+ size_t size; /* inflated size of full object */
};
/*
--git a/oss-fuzz/fuzz-pack-headers.c b/oss-fuzz/fuzz-pack-headers.c
index 150c0f5fa2..ef61ab577c 100644
--- a/oss-fuzz/fuzz-pack-headers.c
+++ b/oss-fuzz/fuzz-pack-headers.c
@@ -6,7 +6,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);
int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)
{
enum object_type type;
- unsigned long len;
+ size_t len;
unpack_object_header_buffer((const unsigned char *)data,
(unsigned long)size, &type, &len);
diff --git a/pack-bitmap.c b/pack-bitmap.c
index f6ec18d83a..f9af8a96bd 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -2270,7 +2270,7 @@ static int try_partial_reuse(struct bitmap_index *bitmap_git,
{
off_t delta_obj_offset;
enum object_type type;
- unsigned long size;
+ size_t size;
if (pack_pos >= pack->p->num_objects)
return -1; /* not actually in the pack */
diff --git a/pack-check.c b/pack-check.c
index 79992bb509..2792f34d25 100644
--- a/pack-check.c
+++ b/pack-check.c
@@ -110,7 +110,7 @@ static int verify_packfile(struct repository *r,
void *data;
struct object_id oid;
enum object_type type;
- unsigned long size;
+ size_t size;
off_t curpos;
int data_valid;
@@ -143,7 +143,9 @@ static int verify_packfile(struct repository *r,
data = NULL;
data_valid = 0;
} else {
- data = unpack_entry(r, p, entries[i].offset, &type, &size);
+ unsigned long sz;
+ data = unpack_entry(r, p, entries[i].offset, &type, &sz);
+ size = sz;
data_valid = 1;
}
diff --git a/packfile.c b/packfile.c
index b012d648ad..fdae91dd11 100644
--- a/packfile.c
+++ b/packfile.c
@@ -1133,7 +1133,7 @@ out:
}
unsigned long unpack_object_header_buffer(const unsigned char *buf,
- unsigned long len, enum object_type *type, unsigned long *sizep)
+ unsigned long len, enum object_type *type, size_t *sizep)
{
unsigned shift;
size_t size, c;
@@ -1144,7 +1144,11 @@ unsigned long unpack_object_header_buffer(const unsigned char *buf,
size = c & 15;
shift = 4;
while (c & 0x80) {
- if (len <= used || (bitsizeof(long) - 7) < shift) {
+ /*
+ * Each continuation byte adds 7 bits. Ensure shift won't
+ * overflow size_t (use size_t not long for 64-bit on Windows).
+ */
+ if (len <= used || (bitsizeof(size_t) - 7) < shift) {
error("bad object header");
size = used = 0;
break;
@@ -1153,7 +1157,7 @@ unsigned long unpack_object_header_buffer(const unsigned char *buf,
size = st_add(size, st_left_shift(c & 0x7f, shift));
shift += 7;
}
- *sizep = cast_size_t_to_ulong(size);
+ *sizep = size;
return used;
}
@@ -1215,7 +1219,7 @@ unsigned long get_size_from_delta(struct packed_git *p,
int unpack_object_header(struct packed_git *p,
struct pack_window **w_curs,
off_t *curpos,
- unsigned long *sizep)
+ size_t *sizep)
{
unsigned char *base;
unsigned long left;
@@ -1367,7 +1371,7 @@ static enum object_type packed_to_object_type(struct repository *r,
while (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {
off_t base_offset;
- unsigned long size;
+ size_t size;
/* Push the object we're going to leave behind */
if (poi_stack_nr >= poi_stack_alloc && poi_stack == small_poi_stack) {
poi_stack_alloc = alloc_nr(poi_stack_nr);
@@ -1586,7 +1590,7 @@ static int packed_object_info_with_index_pos(struct packed_git *p, off_t obj_off
uint32_t *maybe_index_pos, struct object_info *oi)
{
struct pack_window *w_curs = NULL;
- unsigned long size;
+ size_t size;
off_t curpos = obj_offset;
enum object_type type = OBJ_NONE;
uint32_t pack_pos;
@@ -1778,7 +1782,7 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,
struct pack_window *w_curs = NULL;
off_t curpos = obj_offset;
void *data = NULL;
- unsigned long size;
+ size_t size;
enum object_type type;
struct unpack_entry_stack_ent small_delta_stack[UNPACK_ENTRY_STACK_PREALLOC];
struct unpack_entry_stack_ent *delta_stack = small_delta_stack;
@@ -1943,8 +1947,10 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,
(uintmax_t)curpos, p->pack_name);
data = NULL;
} else {
+ unsigned long sz;
data = patch_delta(base, base_size, delta_data,
- delta_size, &size);
+ delta_size, &sz);
+ size = sz;
/*
* We could not apply the delta; warn the user, but
@@ -2929,7 +2935,7 @@ int packfile_read_object_stream(struct odb_read_stream **out,
struct odb_packed_read_stream *stream;
struct pack_window *window = NULL;
enum object_type in_pack_type;
- unsigned long size;
+ size_t size;
in_pack_type = unpack_object_header(pack, &window, &offset, &size);
unuse_pack(&window);
diff --git a/packfile.h b/packfile.h
index 9b647da7dd..49d6bdecf6 100644
--- a/packfile.h
+++ b/packfile.h
@@ -456,9 +456,9 @@ off_t find_pack_entry_one(const struct object_id *oid, struct packed_git *);
int is_pack_valid(struct packed_git *);
void *unpack_entry(struct repository *r, struct packed_git *, off_t, enum object_type *, unsigned long *);
-unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);
+unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, size_t *sizep);
unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);
-int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);
+int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, size_t *);
off_t get_delta_base(struct packed_git *p, struct pack_window **w_curs,
off_t *curpos, enum object_type type,
off_t delta_obj_offset);
--
gitgitgadget
^ permalink raw reply related
* [PATCH v2 02/11] git-zlib: handle data streams larger than 4GB
From: Johannes Schindelin via GitGitGadget @ 2026-05-04 17:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Torsten Bögershausen, Jeff King,
Johannes Schindelin, Johannes Schindelin
In-Reply-To: <pull.2102.v2.git.1777914508.gitgitgadget@gmail.com>
From: Johannes Schindelin <johannes.schindelin@gmx.de>
On Windows, zlib's `uLong` type is 32-bit even on 64-bit systems. When
processing data streams larger than 4GB, the `total_in` and `total_out`
fields in zlib's `z_stream` structure wrap around, which caused the
sanity checks in `zlib_post_call()` to trigger `BUG()` assertions.
The git_zstream wrapper now tracks its own 64-bit totals rather than
copying them from zlib. The sanity checks compare only the low bits,
using `maximum_unsigned_value_of_type(uLong)` to mask appropriately for
the platform's `uLong` size.
This is based on work by LordKiRon in git-for-windows#6076.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
git-zlib.c | 25 +++++++++++++++++--------
git-zlib.h | 4 ++--
object-file.c | 2 +-
3 files changed, 20 insertions(+), 11 deletions(-)
diff --git a/git-zlib.c b/git-zlib.c
index df9604910e..b91cb323ae 100644
--- a/git-zlib.c
+++ b/git-zlib.c
@@ -30,6 +30,9 @@ static const char *zerr_to_string(int status)
*/
/* #define ZLIB_BUF_MAX ((uInt)-1) */
#define ZLIB_BUF_MAX ((uInt) 1024 * 1024 * 1024) /* 1GB */
+
+/* uLong is 32-bit on Windows, even on 64-bit systems */
+#define ULONG_MAX_VALUE maximum_unsigned_value_of_type(uLong)
static inline uInt zlib_buf_cap(unsigned long len)
{
return (ZLIB_BUF_MAX < len) ? ZLIB_BUF_MAX : len;
@@ -39,31 +42,37 @@ static void zlib_pre_call(git_zstream *s)
{
s->z.next_in = s->next_in;
s->z.next_out = s->next_out;
- s->z.total_in = s->total_in;
- s->z.total_out = s->total_out;
+ s->z.total_in = (uLong)(s->total_in & ULONG_MAX_VALUE);
+ s->z.total_out = (uLong)(s->total_out & ULONG_MAX_VALUE);
s->z.avail_in = zlib_buf_cap(s->avail_in);
s->z.avail_out = zlib_buf_cap(s->avail_out);
}
static void zlib_post_call(git_zstream *s, int status)
{
- unsigned long bytes_consumed;
- unsigned long bytes_produced;
+ size_t bytes_consumed;
+ size_t bytes_produced;
bytes_consumed = s->z.next_in - s->next_in;
bytes_produced = s->z.next_out - s->next_out;
- if (s->z.total_out != s->total_out + bytes_produced)
+ /*
+ * zlib's total_out/total_in are uLong which may wrap for >4GB.
+ * We track our own totals and verify only the low bits match.
+ */
+ if ((s->z.total_out & ULONG_MAX_VALUE) !=
+ ((s->total_out + bytes_produced) & ULONG_MAX_VALUE))
BUG("total_out mismatch");
/*
* zlib does not update total_in when it returns Z_NEED_DICT,
* causing a mismatch here. Skip the sanity check in that case.
*/
if (status != Z_NEED_DICT &&
- s->z.total_in != s->total_in + bytes_consumed)
+ (s->z.total_in & ULONG_MAX_VALUE) !=
+ ((s->total_in + bytes_consumed) & ULONG_MAX_VALUE))
BUG("total_in mismatch");
- s->total_out = s->z.total_out;
- s->total_in = s->z.total_in;
+ s->total_out += bytes_produced;
+ s->total_in += bytes_consumed;
/* zlib-ng marks `next_in` as `const`, so we have to cast it away. */
s->next_in = (unsigned char *) s->z.next_in;
s->next_out = s->z.next_out;
diff --git a/git-zlib.h b/git-zlib.h
index 0e66fefa8c..44380e8ad3 100644
--- a/git-zlib.h
+++ b/git-zlib.h
@@ -7,8 +7,8 @@ typedef struct git_zstream {
struct z_stream_s z;
unsigned long avail_in;
unsigned long avail_out;
- unsigned long total_in;
- unsigned long total_out;
+ size_t total_in;
+ size_t total_out;
unsigned char *next_in;
unsigned char *next_out;
} git_zstream;
diff --git a/object-file.c b/object-file.c
index 2acc9522df..086b2b65ff 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1118,7 +1118,7 @@ int odb_source_loose_write_stream(struct odb_source *source,
} while (ret == Z_OK || ret == Z_BUF_ERROR);
if (stream.total_in != len + hdrlen)
- die(_("write stream object %ld != %"PRIuMAX), stream.total_in,
+ die(_("write stream object %"PRIuMAX" != %"PRIuMAX), (uintmax_t)stream.total_in,
(uintmax_t)len + hdrlen);
/*
--
gitgitgadget
^ permalink raw reply related
* [PATCH v2 01/11] index-pack, unpack-objects: use size_t for object size
From: Johannes Schindelin via GitGitGadget @ 2026-05-04 17:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Torsten Bögershausen, Jeff King,
Johannes Schindelin, Johannes Schindelin
In-Reply-To: <pull.2102.v2.git.1777914508.gitgitgadget@gmail.com>
From: Johannes Schindelin <johannes.schindelin@gmx.de>
When unpacking objects from a packfile, the object size is decoded
from a variable-length encoding. On platforms where unsigned long is
32-bit (such as Windows, even in 64-bit builds), the shift operation
overflows when decoding sizes larger than 4GB. The result is a
truncated size value, causing the unpacked object to be corrupted or
rejected.
Fix this by changing the size variable to size_t, which is 64-bit on
64-bit platforms, and ensuring the shift arithmetic occurs in 64-bit
space.
This was originally authored by LordKiRon <https://github.com/LordKiRon>,
who preferred not to reveal their real name and therefore agreed that I
take over authorship.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
builtin/index-pack.c | 9 +++++----
builtin/unpack-objects.c | 5 +++--
2 files changed, 8 insertions(+), 6 deletions(-)
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index ca7784dc2c..cc660582e9 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -37,7 +37,7 @@ static const char index_pack_usage[] =
struct object_entry {
struct pack_idx_entry idx;
- unsigned long size;
+ size_t size;
unsigned char hdr_size;
signed char type;
signed char real_type;
@@ -469,7 +469,7 @@ static int is_delta_type(enum object_type type)
return (type == OBJ_REF_DELTA || type == OBJ_OFS_DELTA);
}
-static void *unpack_entry_data(off_t offset, unsigned long size,
+static void *unpack_entry_data(off_t offset, size_t size,
enum object_type type, struct object_id *oid)
{
static char fixed_buf[8192];
@@ -524,7 +524,8 @@ static void *unpack_raw_entry(struct object_entry *obj,
struct object_id *oid)
{
unsigned char *p;
- unsigned long size, c;
+ size_t size;
+ unsigned long c;
off_t base_offset;
unsigned shift;
void *data;
@@ -542,7 +543,7 @@ static void *unpack_raw_entry(struct object_entry *obj,
p = fill(1);
c = *p;
use(1);
- size += (c & 0x7f) << shift;
+ size += ((size_t)c & 0x7f) << shift;
shift += 7;
}
obj->size = size;
diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
index e01cf6e360..59a36c2481 100644
--- a/builtin/unpack-objects.c
+++ b/builtin/unpack-objects.c
@@ -533,7 +533,8 @@ static void unpack_one(unsigned nr)
{
unsigned shift;
unsigned char *pack;
- unsigned long size, c;
+ size_t size;
+ unsigned long c;
enum object_type type;
obj_list[nr].offset = consumed_bytes;
@@ -548,7 +549,7 @@ static void unpack_one(unsigned nr)
pack = fill(1);
c = *pack;
use(1);
- size += (c & 0x7f) << shift;
+ size += ((size_t)c & 0x7f) << shift;
shift += 7;
}
--
gitgitgadget
^ permalink raw reply related
* [PATCH v2 00/11] Handle cloning of objects larger than 4GB on Windows
From: Johannes Schindelin via GitGitGadget @ 2026-05-04 17:08 UTC (permalink / raw)
To: git
Cc: Derrick Stolee, Torsten Bögershausen, Jeff King,
Johannes Schindelin
In-Reply-To: <pull.2102.git.1777393580.gitgitgadget@gmail.com>
On Windows, unsigned long is 32-bit even on 64-bit systems. This causes
multiple problems when Git handles objects larger than 4GB. This patch
series is a very targeted fix for a very early part of the problem: it
addresses the most fundamental truncation points that prevent a >4GB object
from surviving a clone at all.
Specifically, this fixes:
* zlib's uLong wrapping and triggering BUG() assertions in the git_zstream
wrapper
* Object sizes being truncated in pack streaming, delta headers, and
index-pack/unpack-objects
* pack-objects re-encoding reused pack entries with a truncated size,
producing corrupt packs on the wire
Many other code paths still use unsigned long for object sizes (e.g.,
cat-file -s, object_info.sizep, the delta machinery) and will need their own
conversions. This series does not attempt to fix those.
Based on work by @LordKiRon in git-for-windows/git#6076.
The last two commits add a test helper that synthesizes a pack with a >4GB
blob and regression tests that clone it via both the unpack-objects and
index-pack code paths using file:// transport.
Changes since v1:
* dramatically accelerated the test helper that generates 4GB pack files,
via two separate strategies:
1. using the "unsafe" SHA-1 for the blob OID computation.
2. using pre-computed "Lego blocks" to construct the 4GB packs needed in
the test cases, where the size (and therefore the involved OIDs) are
well-known in advance.
* even with these improvements, the actual git clone is still slow (of
course, because it cannot use any of those shortcuts), therefore the
tests are marked as EXPENSIVE.
* to exercise those tests nevertheless, the last patch lets all EXPENSIVE
test cases be run for the integration branches other than seen.
Johannes Schindelin (11):
index-pack, unpack-objects: use size_t for object size
git-zlib: handle data streams larger than 4GB
odb, packfile: use size_t for streaming object sizes
delta, packfile: use size_t for delta header sizes
test-tool: add a helper to synthesize large packfiles
t5608: add regression test for >4GB object clone
test-tool synthesize: use the unsafe hash for speed
test-tool synthesize: precompute pack for 4 GiB + 1
test-tool synthesize: add precomputed SHA-256 pack for 4 GiB + 1
t5608: mark >4GB tests as EXPENSIVE
ci: run expensive tests on push builds to integration branches
Makefile | 1 +
builtin/index-pack.c | 9 +-
builtin/pack-objects.c | 23 +-
builtin/unpack-objects.c | 5 +-
ci/lib.sh | 9 +
compat/zlib-compat.h | 2 +
delta.h | 14 +-
git-zlib.c | 25 +-
git-zlib.h | 4 +-
object-file.c | 12 +-
odb/streaming.c | 13 +-
odb/streaming.h | 2 +-
oss-fuzz/fuzz-pack-headers.c | 2 +-
pack-bitmap.c | 2 +-
pack-check.c | 6 +-
packfile.c | 57 ++--
packfile.h | 4 +-
t/helper/meson.build | 1 +
t/helper/test-synthesize.c | 541 +++++++++++++++++++++++++++++++++++
t/helper/test-tool.c | 1 +
t/helper/test-tool.h | 1 +
t/t5608-clone-2gb.sh | 37 +++
22 files changed, 718 insertions(+), 53 deletions(-)
create mode 100644 t/helper/test-synthesize.c
base-commit: 94f057755b7941b321fd11fec1b2e3ca5313a4e0
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2102%2Fdscho%2Ffix-large-clones-on-windows-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2102/dscho/fix-large-clones-on-windows-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/2102
Range-diff vs v1:
1: dc660106ea = 1: dc660106ea index-pack, unpack-objects: use size_t for object size
2: 92f4327b1f = 2: 92f4327b1f git-zlib: handle data streams larger than 4GB
3: 3a539061c5 = 3: 3a539061c5 odb, packfile: use size_t for streaming object sizes
4: 3274cba862 = 4: 3274cba862 delta, packfile: use size_t for delta header sizes
5: afa74a3a2b = 5: afa74a3a2b test-tool: add a helper to synthesize large packfiles
6: a3019888d8 = 6: a3019888d8 t5608: add regression test for >4GB object clone
-: ---------- > 7: 859e93e7a9 test-tool synthesize: use the unsafe hash for speed
-: ---------- > 8: 29b9a74e91 test-tool synthesize: precompute pack for 4 GiB + 1
-: ---------- > 9: 8e6e720804 test-tool synthesize: add precomputed SHA-256 pack for 4 GiB + 1
-: ---------- > 10: 5b44410b2f t5608: mark >4GB tests as EXPENSIVE
-: ---------- > 11: 1eaaa7fad7 ci: run expensive tests on push builds to integration branches
--
gitgitgadget
^ permalink raw reply
* Re: [PATCH 6/6] t5608: add regression test for >4GB object clone
From: Johannes Schindelin @ 2026-05-04 17:07 UTC (permalink / raw)
To: Derrick Stolee; +Cc: Jeff King, Johannes Schindelin via GitGitGadget, git
In-Reply-To: <ed226571-a095-456b-9d9e-bcc545d7ddfb@gmail.com>
Hi Stolee & Jeff,
On Sun, 3 May 2026, Derrick Stolee wrote:
> On 5/1/2026 2:38 AM, Jeff King wrote:
> > On Wed, Apr 29, 2026 at 09:34:21AM -0400, Derrick Stolee wrote:
>
> >>> +test_expect_success SIZE_T_IS_64BIT 'set up repo with >4GB object' '
> >>
> >> Your prereq here prevents it from running on 32-bit builds, which is
> >> good. However, I wonder if it would be worth also specifying these
> >> tests as expensive. It's less likely that these layers will be touched
> >> often, so it should be enough to run these on major occasions, such as
> >> testing a release candidate.
> >
> > I think it is already skipped in most cases, because t5608 requires the
> > GIT_TEST_CLONE_2GB environment variable be set. Arguably it should just
> > be using EXPENSIVE, too, as I do not think there is much value in having
> > individual flags for all of the expensive tests. I think that test just
> > predates the modern prereq system entirely.
>
> Thanks for the extra details here! That helps avoid the issues that I
> was thinking about, but maybe doubling-down and adding EXPENSIVE is
> still worth it.
Indeed. The `GIT_TEST_CLONE_2GB` flag is set for all CI jobs:
https://gitlab.com/git-scm/git/-/blob/v2.54.0/ci/lib.sh#L314
So I've tried to accelerate it.
First, by using the "unsafe" SHA-1 (because we're in no danger here, we
generate the data ourselves). That helped some, but unfortunately the most
efficient implementation (OpenSSL) is out of bounds for us because we
cannot enable its use in the default configuration for licensing reasons:
OpenSSL's license and Git's GPLv2 are fundamentally incompatible with each
other, and even though Linus' intention with
https://gitlab.com/git-scm/git/-/blob/e83c5163316f89bfbde7d9ab23ca2e25604af290/Makefile#L11
was clearly to allow linking to OpenSSL (actually, requiring it), there is
one Git contributor I spoke to who flatly stated that they'd block every
attempt to add an exception to Git's license retroactively to allow
linking to OpenSSL (at least for distribution, which is when the GPLv2
would kick in). So that's that.
Second, I added shortcuts for the 4GiB+1 size exercised in the added test
cases. That did help! But only the generation time of those packfiles is
helpd by that. The `git clone` that is tested is still awfully slow, and
that _cannot_ be worked around in the same ways.
Therefore, I ended up marking the test cases as `EXPENSIVE`.
To allow them to be run regularly anyway, I added a final patch to the
series that lets the CI runs on the integration branches (other than
`seen`) run all `EXPENSIVE` test cases. One could argue that this patch
should be split out, and I'm open to it, but in the interest of keeping
the time I am working on this patch series _somewhat_ closer to what could
be called reasonable, I'll just keep it in the patch series for now,
hedging for the possibility that maybe nobody objects?
Ciao,
Johannes
> >> I suppose this also is a question for Junio and our process for
> >> validating releases. Do we have a certain cadence where we run the
> >> expensive tests? What has been our threshold for hiding a test case
> >> behind the expensive label?
> >
> > AFAIK the labeling of expensive things is mostly ad-hoc, and nobody is
> > systematically running them. Likewise for the t/perf tests, which are
> > super expensive but do (very occasionally) turn up interesting
> > regressions.
> I used to be more diligent about running the performance tests myself
> around release windows. The EXPENSIVE tests would also be good to do
> on rc0. I will contemplate how to put this into my routine.
>
> Thanks,
> -Stolee
>
>
>
^ permalink raw reply
* Question on Clean/Smudge Infrastructure
From: rsbecker @ 2026-05-04 16:57 UTC (permalink / raw)
To: git
Hi Git,
I have a edge use case that I would like to ask about.
Given a directory with a large number, say 100, text files, and a few
scattered
binary files - specified in .gitattributes as binary, what does clean smudge
do
with the binary files if they match the filter specification pattern? Are
they
ignored or processed. I am not sure that passing binary via stdin is
necessarily
portable. However, I would like to be able to explicitly ignore the binary
files
in my clean/smudge filters - either by doing a copy stdin/stdout (as I said,
probably
not portable), or sending a non-zero exit code, or some other mechanism.
The root of the use case is that the directory is subject to significant
changes
over time, and errors are sneaking in when people forget to update
.gitattributes
or name the files incorrectly. I would like to make their situation more
stable
to errors.
Thanks,
Randall
--
Brief whoami: NonStop&UNIX developer since approximately
UNIX(421664400)
NonStop(211288444200000000)
-- In real life, I talk too much.
^ permalink raw reply
* Git trims the last character of content from remotes
From: Hugo Osvaldo Barrera @ 2026-05-04 17:01 UTC (permalink / raw)
To: git
Hi all,
When I push content to GitLab, the remote server sends back some text which git
then prints to stderr:
remote:
remote: To create a merge request for zk, visit:
remote: https://gitlab.alpinelinux.org/WhyNotHugo/aports/-/merge_requests/new?merge_request%5Bsource_branch%5D=zk
remote:
When the width of a whole line is the same as my terminal width, the last digit
gets trimmed off. E.g.: if I resize my terminal for the above to fix exactly,
and re-run the same command, git prints:
remote: https://gitlab.alpinelinux.org/WhyNotHugo/aports/-/merge_requests/new?merge_request%5Bsource_branch%5D=z
From what I can tell, sideband.c prints ANSI_SUFFIX = "\033[K", this escape
sequence being "clear the line from the current position until the end of the
line", and this is the root cause of the issue.
When piping to cat or to a file, this sequence is not printed, so the output is
fine.
Is this a bug?
Thanks,
PS: please CC me, as I am not subscribed to the list.
--
Hugo
^ permalink raw reply
* git hogs the CPU, RAM and storage despite its config
From: jean-christophe manciot @ 2026-05-04 15:27 UTC (permalink / raw)
To: git
What did you do before the bug happened? (Steps to reproduce your issue)
many:
git add -v --force -- "${file_path}/${file_name}"
git commit -v -m "${full_commit_message}" -o -- "${file_path}/${file_name}"
What did you expect to happen? (Expected behavior)
I expected git to respect the configuration (.git/config):
[core]
repositoryformatversion = 0
filemode = false
bare = false
logallrefupdates = true
fsmonitor = true
untrackedcache = true
[remote "origin"]
url = git@examle.com:ppa
fetch = +refs/heads/*:refs/remotes/origin/*
[branch]
autosetuprebase = always
[branch "main"]
remote = origin
merge = refs/heads/main
rebase = true
[feature]
manyFiles = true
[fetch]
writeCommitGraph = true
[gc]
auto = 0
[pack]
threads = 1
windowMemory = 1g
I expected git to use maximum one thread for packing and I'm surprised
it even tried to perform packing as gc.auto was disabled.
What happened instead? (Actual behavior)
Instead, it used all the threads it could find (28 out of 32),
depriving the whole server of CPU, RAM and storage as the tmp files
kept piling up.
What's different between what you expected and what actually happened?
Uncontrollable use of CPU threads, RAM and storage
Anything else you want to add:
All the threads were running:
git pack-objects --local --delta-base-offset --honor-pack-keep
.git/objects/pack/.tmp-<number>-pack
Please review the rest of the bug report below.
You can delete any lines you don't wish to share.
[System Info]
git version:
git version 2.51.0
cpu: x86_64
no commit associated with this build
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh
libcurl: 8.14.1
zlib: 1.3.1
SHA-1: SHA1_DC
SHA-256: SHA256_BLK
default-ref-format: files
default-hash: sha1
uname: Linux 6.17.0-23-generic #23-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr
11 23:29:57 UTC 2026 x86_64
compiler info: gnuc: 15.2
libc info: glibc: 2.42
$SHELL (typically, interactive shell): /bin/bash
[Enabled Hooks]
--
Jean-Christophe
^ permalink raw reply
* Re: [PATCH v3 1/1] git-gui: handle missing worktree and separated gitdir
From: Mark Levedahl @ 2026-05-04 15:13 UTC (permalink / raw)
To: Johannes Sixt; +Cc: git, Shroom Moo
In-Reply-To: <73b99b54-1d39-45c1-bd06-26ac1008fddb@kdbg.org>
On 5/3/26 4:53 AM, Johannes Sixt wrote:
> I would not call the use of --is-bar-repository instead of
> --is-inside-git-dir an error, just a choice that has been made. In
> particular, when the startup directory is named '.git' and is not marked
> as bare, then its parent directory can very reasonably be taken as its
> worktree. (That's how things worked before --show-toplevel was used.) If
> the check is for --is-inside-git-dir, this treatment would be ruled out
> early.
>
Whether being in a gitdir is ok, or a worktree required, is of fundamental importance and
is not explicitly checked now. This is my issue. (Whether the repo is bare, or embedded in
a worktree, is relevant only when automatically fixing a user error.)
> But perhaps there is a simpler solution: Let's present an error if
> --show-toplevel fails except in the case where the startup directory is
> named '.git' (and is a valid Git repository) and is not bare (then the
> worktree is the parent). I insist in this exception, because this
> use-case was considered important in the past (87cd09f43e56 "git-gui:
> work from the .git dir", 2010-01-23).
>
> -- Hannes
>
This would not fix gitk's blame / browse from a gitdir, and I don't really see a one or
two line fix as being adequate.
git-gui sets GIT_WORK_TREE and GIT_DIR at startup. GIT_DIR passes my simple tests, but
mishandles GIT_WORK_TREE.
I expect these two invocations to be equivalent, both starting git-gui in the worktree
'/some/path':
GIT_WORK_TREE=/some/path git gui
git -C /some/path gui
But, the GIT_WORK_TREE approach:
works as I expect ONLY when the current directory is a valid worktree
when started from a gitdir, uses that gitdir in conjunction with the requested worktree
when started from an uncontrolled directory, shows the repository picker.
The git -C approach is indifferent to the current directory, of course.
GIT_WORK_TREE enters much too late in the process, and rather should handled first:
if GIT_WORK_TREE is in the environment, cd to that first. Throw an error if that
directory is not a valid worktree.
I don't actually understand the use case of defining GIT_DIR or GIT_WORK_TREE to git gui,
and I wonder what other bugs are lurking... maybe the better approach is to just abort if
GIT_DIR or GIT_WORK_TREE are defined?
Mark
^ permalink raw reply
* [PATCH v5 1/1] git-gui: restructure repository startup
From: Shroom Moo @ 2026-05-04 14:59 UTC (permalink / raw)
To: git; +Cc: Johannes Sixt, Mark Levedahl, Shroom Moo
In-Reply-To: <tencent_277823B7C5D69914E168E5679A907C655606@qq.com>
When git-gui is started inside a .git directory of a non-bare
repository, it should treat the parent directory as the worktree,
as it did before commit 2d92ab32fd (rev-parse: make --show-toplevel
without a worktree an error, 2019-11-19). However, a bare repository
or a separated gitdir without a worktree must be rejected early.
Protect the previously unguarded calls to `git rev-parse
--show-object-format` and `--show-toplevel`. Restructure the startup
sequence to:
- Check for a bare repository right after loading the config. If the
repository is bare and the current subcommand does not allow bare
repos (e.g. normal commit mode), show "Cannot use bare repository"
and exit.
- When `rev-parse --show-toplevel` fails and the repository is
non-bare, the gitdir path ends with ".git", and we are inside that
gitdir, use the parent directory as the worktree. This preserves
the ability to start git-gui from within a regular repository’s
.git directory, which was intentionally supported since 87cd09f43e56
(git-gui: work from the .git dir, 2010-01-23).
- Otherwise, show a descriptive error and exit.
- Wrap `rev-parse --show-object-format` in a catch to avoid a crash
when the repository configuration is broken (e.g. core.worktree
pointing to an invalid path).
Also removes the old `_prefix`‑based fallback that computed a relative
path to the worktree top from a subdirectory, and the unconditional
`[file dirname $_gitdir]` guess. Both are unnecessary now that
`rev‑parse --show‑toplevel` directly provides the absolute top‑level
path and we can `cd` to it. The guess is further unsafe in
multi‑worktree setups, where a gitdir may have more than one worktree.
The only remaining fallback is the explicit “.git directory” rule for
non‑bare repositories, which mirrors the historical behaviour.
This fixes the fatal Tcl error when the working tree is missing, while
keeping the .git startup feature and avoiding any automatic directory
switching that could be dangerous in multi‑worktree setups.
Signed-off-by: Shroom Moo <egg_mushroomcow@foxmail.com>
---
git-gui/git-gui.sh | 72 +++++++++++++++++++++++++++++-----------------
1 file changed, 45 insertions(+), 27 deletions(-)
diff --git a/git-gui/git-gui.sh b/git-gui/git-gui.sh
index 23fe76e498..c06d85b8d9 100755
--- a/git-gui/git-gui.sh
+++ b/git-gui/git-gui.sh
@@ -1129,7 +1129,8 @@ if {[catch {
}]
&& [catch {
# beware that from the .git dir this sets _gitdir to .
- # and _prefix to the empty string
+ # and _prefix to the empty string; this is handled by
+ # the startup safety checks below
set _gitdir [git rev-parse --git-dir]
set _prefix [git rev-parse --show-prefix]
} err]} {
@@ -1142,8 +1143,20 @@ if {[catch {
set picked 1
}
+if {![file isdirectory $_gitdir]} {
+ catch {wm withdraw .}
+ error_popup [strcat
+ [mc "Git directory not found:"] "\n\n$_gitdir\n\n" \
+ [mc "Please ensure GIT_DIR points to a valid Git repository"]]
+ exit 1
+}
+
# Use object format as hash algorithm (either "sha1" or "sha256")
-set hashalgorithm [git rev-parse --show-object-format]
+if {[catch {set hashalgorithm [git rev-parse --show-object-format]} err]} {
+ catch {wm withdraw .}
+ error_popup [strcat [mc "Failed to determine hash algorithm:"] "\n\n$err"]
+ exit 1
+}
if {$hashalgorithm eq "sha1"} {
set hashlength 40
} elseif {$hashalgorithm eq "sha256"} {
@@ -1160,46 +1173,50 @@ if {$_gitdir eq "."} {
set _gitdir [pwd]
}
-if {![file isdirectory $_gitdir]} {
- catch {wm withdraw .}
- error_popup [strcat [mc "Git directory not found:"] "\n\n$_gitdir"]
- exit 1
-}
# _gitdir exists, so try loading the config
load_config 0
apply_config
-set _gitworktree [git rev-parse --show-toplevel]
+# Handle bare repository early: if not allowed, abort
+if {[is_bare] && ![is_enabled bare]} {
+ catch {wm withdraw .}
+ error_popup [strcat [mc "Cannot use bare repository:"] "\n\n" [file normalize $_gitdir]]
+ exit 1
+}
-if {$_prefix ne {}} {
- if {$_gitworktree eq {}} {
- regsub -all {[^/]+/} $_prefix ../ cdup
- } else {
- set cdup $_gitworktree
- }
- if {[catch {cd $cdup} err]} {
- catch {wm withdraw .}
- error_popup [strcat [mc "Cannot move to top of working directory:"] "\n\n$err"]
- exit 1
+# Determine the working tree
+if {[is_bare] && [is_enabled bare]} {
+ set _gitworktree {}
+} else {
+ if {[catch {set _gitworktree [git rev-parse --show-toplevel]} err]} {
+ # If we are inside a .git directory of a non-bare repo,
+ # the worktree is the parent directory
+ set inside_gitdir 0
+ catch {set inside_gitdir [git rev-parse --is-inside-git-dir]}
+ if {![is_bare] && $inside_gitdir eq {true} && [file tail [file normalize $_gitdir]] eq {.git}} {
+ set _gitworktree [file normalize [file dirname $_gitdir]]
+ } else {
+ catch {wm withdraw .}
+ error_popup [strcat [mc "Cannot determine working tree:"] "\n\n$err"]
+ exit 1
+ }
}
- set _gitworktree [pwd]
- unset cdup
-} elseif {![is_enabled bare]} {
- if {[is_bare]} {
+
+ if {$_gitworktree eq {}} {
catch {wm withdraw .}
- error_popup [strcat [mc "Cannot use bare repository:"] "\n\n$_gitdir"]
+ error_popup [mc "Cannot determine working tree (unexpected empty result)"]
exit 1
}
- if {$_gitworktree eq {}} {
- set _gitworktree [file dirname $_gitdir]
- }
+
if {[catch {cd $_gitworktree} err]} {
catch {wm withdraw .}
- error_popup [strcat [mc "No working directory"] " $_gitworktree:\n\n$err"]
+ error_popup [strcat [mc "Cannot move to working directory:"] "\n\n$err"]
exit 1
}
set _gitworktree [pwd]
}
+
+# Derive a human-readable repository name
set _reponame [file split [file normalize $_gitdir]]
if {[lindex $_reponame end] eq {.git}} {
set _reponame [lindex $_reponame end-1]
@@ -1207,6 +1224,7 @@ if {[lindex $_reponame end] eq {.git}} {
set _reponame [lindex $_reponame end]
}
+# Export the final paths
set env(GIT_DIR) $_gitdir
set env(GIT_WORK_TREE) $_gitworktree
--
2.52.0.windows.1
^ permalink raw reply related
* Re: What's cooking in git.git (May 2026, #01)
From: Patrick Steinhardt @ 2026-05-04 14:18 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <xmqqmryhtar8.fsf@gitster.g>
On Sun, May 03, 2026 at 12:47:23PM +0900, Junio C Hamano wrote:
> * kn/refs-generic-helpers (2026-04-27) 9 commits
> - refs: use peeled tag values in reference backends
> - refs: add peeled object ID to the `ref_update` struct
> - refs: move object parsing to the generic layer
> - update-ref: handle rejections while adding updates
> - update-ref: move `print_rejected_refs()` up
> - refs: return `ref_transaction_error` from `ref_transaction_update()`
> - refs: extract out reflog config to generic layer
> - refs: introduce `ref_store_init_options`
> - refs: remove unused typedef 'ref_transaction_commit_fn'
>
> Refactor service routines in the ref subsystem backends.
>
> Will merge to 'next'?
> source: <20260427-refs-move-to-generic-layer-v3-0-e4638dfb7897@gmail.com>
I think [1] is something that may still want to be addressed.
> * js/maintenance-fix-deadlock-on-win10 (2026-04-28) 2 commits
> - maintenance(geometric): do release the `.idx` files before repacking
> - mingw: optionally use legacy (non-POSIX) delete semantics
>
> To help Windows 10 installations, avoid removing files whose
> contents are still mmap()'ed.
>
> Will merge to 'next'?
> source: <pull.2103.git.1777380768.gitgitgadget@gmail.com>
I've had a minor nit [2], but other than that this looks good to me.
> * sg/t6112-unwanted-tilde-expansion-fix (2026-04-21) 1 commit
> - t6112: avoid tilde expansion
>
> Test fix.
>
> Will merge to 'next'?
> source: <20260421192132.51172-1-szeder.dev@gmail.com>
This fix looks sensible to me.
> * ps/odb-in-memory (2026-04-10) 18 commits
> - t/unit-tests: add tests for the in-memory object source
> - odb: generic in-memory source
> - odb/source-inmemory: stub out remaining functions
> - odb/source-inmemory: implement `freshen_object()` callback
> - odb/source-inmemory: implement `count_objects()` callback
> - odb/source-inmemory: implement `find_abbrev_len()` callback
> - odb/source-inmemory: implement `for_each_object()` callback
> - odb/source-inmemory: convert to use oidtree
> - oidtree: add ability to store data
> - cbtree: allow using arbitrary wrapper structures for nodes
> - odb/source-inmemory: implement `write_object_stream()` callback
> - odb/source-inmemory: implement `write_object()` callback
> - odb/source-inmemory: implement `read_object_stream()` callback
> - odb/source-inmemory: implement `read_object_info()` callback
> - odb: fix unnecessary call to `find_cached_object()`
> - odb/source-inmemory: implement `free()` callback
> - odb: introduce "in-memory" source
> - Merge branch 'jt/odb-transaction-write' into ps/odb-in-memory
> (this branch uses jt/odb-transaction-write.)
>
> Add a new odb "in-memory" source that is meant to only hold
> tentative objects (like the virtual blob object that represents the
> working tree file used by "git blame").
>
> Will merge to 'next'?
> source: <20260410-b4-pks-odb-source-inmemory-v3-0-22fd0fad58fe@pks.im>
This series should be ready, yes.
> * jt/odb-transaction-write (2026-04-02) 7 commits
> - odb/transaction: make `write_object_stream()` pluggable
> - object-file: generalize packfile writes to use odb_write_stream
> - object-file: avoid fd seekback by checking object size upfront
> - object-file: remove flags from transaction packfile writes
> - odb: update `struct odb_write_stream` read() callback
> - odb/transaction: use pluggable `begin_transaction()`
> - odb: split `struct odb_transaction` into separate header
> (this branch is used by ps/odb-in-memory.)
>
> ODB transaction interface is being reworked to explicitly handle
> object writes.
>
> Will merge to 'next'?
> source: <20260402213220.2651523-1-jltobler@gmail.com>
The latest version looks good to me.
Thanks!
Patrick
[1]: <87v7dagdjk.fsf@toon--20250203-5JQV3.mail-host-address-is-not-set>
[2]: <afipTWyj2zVYYqMz@pks.im>
^ permalink raw reply
* Re: [PATCH 0/2] maintenance(geometric): avoid deadlocks on Windows 10
From: Patrick Steinhardt @ 2026-05-04 14:12 UTC (permalink / raw)
To: Derrick Stolee
Cc: Johannes Schindelin via GitGitGadget, git, Johannes Schindelin
In-Reply-To: <5c358919-0dcf-41c8-bdf7-912c41f77c31@gmail.com>
On Tue, Apr 28, 2026 at 11:01:34AM -0400, Derrick Stolee wrote:
> On 4/28/2026 8:52 AM, Johannes Schindelin via GitGitGadget wrote:
> > On Windows, maintenance_task_geometric_repack() opens pack index files via
> > pack_geometry_init() (which mmap()s the .idx files), then spawns git repack
> > as a child process without setting child.odb_to_close. The parent's mmap()s
> > prevent the child from deleting old .idx files.
> >
> > On Windows 10 builds before the POSIX delete semantics change (between Build
> > 17134.1304 and 18363.657, see https://stackoverflow.com/a/60512798), this
> > results in Unlink of file '.git/objects/pack/pack-<hash>.idx' failed. Should
> > I try again? during fetch-triggered auto-maintenance with the geometric
> > strategy.
> >
> > The fix adds the missing child.odb_to_close = the_repository->objects line,
> > matching all other maintenance tasks.
> >
> > The first commit introduces a GIT_TEST_LEGACY_DELETE environment variable to
> > simulate legacy (pre-POSIX) delete semantics on modern Windows, so the
> > regression test can verify the fix even on Windows 11.
> >
> > This fixes https://github.com/git-for-windows/git/issues/6210.
>
> Thanks for these patches. I reviewed their equivalents in the
> git-for-windows/git fork so I'll give my LGTM here, too.
I've got a single comment on the first patch, but other than that this
series looks good to me. Thanks!
Patrick
^ permalink raw reply
* Re: [PATCH 1/2] mingw: optionally use legacy (non-POSIX) delete semantics
From: Patrick Steinhardt @ 2026-05-04 14:12 UTC (permalink / raw)
To: Johannes Schindelin via GitGitGadget; +Cc: git, Johannes Schindelin
In-Reply-To: <97508e91b62c91b77447dbba39a84770682591a8.1777380768.git.gitgitgadget@gmail.com>
On Tue, Apr 28, 2026 at 12:52:47PM +0000, Johannes Schindelin via GitGitGadget wrote:
> diff --git a/compat/mingw.c b/compat/mingw.c
> index 2023c16db6..04f9aa3922 100644
> --- a/compat/mingw.c
> +++ b/compat/mingw.c
> @@ -449,20 +449,63 @@ static wchar_t *normalize_ntpath(wchar_t *wbuf)
> return wbuf;
> }
>
> +/*
> + * Use SetFileInformationByHandle(FileDispositionInfo) to force legacy
> + * (non-POSIX) delete semantics. On Windows 11, DeleteFileW() uses POSIX
> + * delete semantics internally, allowing deletion even with active
> + * MapViewOfFile views. This helper simulates Windows 10 behavior where
> + * deletion fails if a file mapping exists.
> + *
> + * Returns nonzero on success (like DeleteFileW), 0 on failure.
> + */
> +static int legacy_delete_file(const wchar_t *wpathname)
> +{
> + FILE_DISPOSITION_INFO fdi = { TRUE };
> + DWORD gle;
> + HANDLE h = CreateFileW(wpathname, DELETE,
> + FILE_SHARE_READ | FILE_SHARE_WRITE |
> + FILE_SHARE_DELETE,
> + NULL, OPEN_EXISTING,
> + FILE_FLAG_OPEN_REPARSE_POINT, NULL);
> + if (h == INVALID_HANDLE_VALUE)
> + return 0;
> +
> + if (SetFileInformationByHandle(h, FileDispositionInfo,
> + &fdi, sizeof(fdi))) {
> + CloseHandle(h);
> + return 1;
> + }
> + gle = GetLastError();
> + CloseHandle(h);
> + SetLastError(gle);
> + return 0;
> +}
> +
> +static int try_delete_file(const wchar_t *wpathname, int use_legacy)
> +{
> + if (use_legacy)
> + return legacy_delete_file(wpathname);
> + return DeleteFileW(wpathname);
> +}
> +
> int mingw_unlink(const char *pathname, int handle_in_use_error)
> {
> + static int use_legacy_delete = -1;
> int tries = 0;
> wchar_t wpathname[MAX_PATH];
> if (xutftowcs_path(wpathname, pathname) < 0)
> return -1;
>
> - if (DeleteFileW(wpathname))
> + if (use_legacy_delete < 0)
> + use_legacy_delete = !!getenv("GIT_TEST_LEGACY_DELETE");
Should this use `git_env_bool()`?
Patrick
^ permalink raw reply
* [PATCH v2 4/4] xdiff: reduce the size of array
From: Phillip Wood @ 2026-05-04 14:06 UTC (permalink / raw)
To: git; +Cc: Ezekiel Newren, Junio C Hamano, Phillip Wood
In-Reply-To: <cover.1777903579.git.phillip.wood@dunelm.org.uk>
From: Phillip Wood <phillip.wood@dunelm.org.uk>
When the myers algorithm is selected the input files are pre-processed
to remove any common prefix and suffix and any lines that appear
in only one file. This requires a map to be created between the
lines that are processed by the myers algorithm and the lines in
the original file. That map does not include the common lines at the
beginning and end of the files but the array is allocated to be the
size of the whole file. Move the allocation into xdl_cleanup_records()
where the map is populated and we know how big it needs to be.
Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>
---
xdiff/xprepare.c | 11 ++++-------
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/xdiff/xprepare.c b/xdiff/xprepare.c
index 7a29e5fc474..11bada2608a 100644
--- a/xdiff/xprepare.c
+++ b/xdiff/xprepare.c
@@ -170,12 +170,6 @@ static int xdl_prepare_ctx(unsigned int pass, mmfile_t *mf, long narec, xpparam_
if (!XDL_CALLOC_ARRAY(xdf->changed, xdf->nrec + 2))
goto abort;
-
- if ((XDF_DIFF_ALG(xpp->flags) != XDF_PATIENCE_DIFF) &&
- (XDF_DIFF_ALG(xpp->flags) != XDF_HISTOGRAM_DIFF)) {
- if (!XDL_ALLOC_ARRAY(xdf->reference_index, xdf->nrec + 1))
- goto abort;
- }
xdf->changed += 1;
xdf->nreff = 0;
@@ -283,7 +277,10 @@ static int xdl_cleanup_records(xdlclassifier_t *cf, xdfile_t *xdf1, xdfile_t *xd
* changed[i] should remain false, or become true.
*/
if (!XDL_CALLOC_ARRAY(action1, len1) ||
- !XDL_CALLOC_ARRAY(action2, len2)) {
+ !XDL_CALLOC_ARRAY(action2, len2) ||
+ !XDL_ALLOC_ARRAY(xdf1->reference_index, len1) ||
+ !XDL_ALLOC_ARRAY(xdf2->reference_index, len2))
+ {
ret = -1;
goto cleanup;
}
--
2.54.0.rc1.174.gd833f386ac5.dirty
^ permalink raw reply related
* [PATCH v2 3/4] xprepare: simplify error handling
From: Phillip Wood @ 2026-05-04 14:06 UTC (permalink / raw)
To: git; +Cc: Ezekiel Newren, Junio C Hamano, Phillip Wood
In-Reply-To: <cover.1777903579.git.phillip.wood@dunelm.org.uk>
From: Phillip Wood <phillip.wood@dunelm.org.uk>
If either of the two allocations fail we want to take the same action
so use a single if statement. This saves a few lines and makes it
easier for the next commit to add a couple more allocations.
Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk>
---
xdiff/xprepare.c | 7 ++-----
1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/xdiff/xprepare.c b/xdiff/xprepare.c
index 81de412875a..7a29e5fc474 100644
--- a/xdiff/xprepare.c
+++ b/xdiff/xprepare.c
@@ -282,11 +282,8 @@ static int xdl_cleanup_records(xdlclassifier_t *cf, xdfile_t *xdf1, xdfile_t *xd
* Create temporary arrays that will help us decide if
* changed[i] should remain false, or become true.
*/
- if (!XDL_CALLOC_ARRAY(action1, len1)) {
- ret = -1;
- goto cleanup;
- }
- if (!XDL_CALLOC_ARRAY(action2, len2)) {
+ if (!XDL_CALLOC_ARRAY(action1, len1) ||
+ !XDL_CALLOC_ARRAY(action2, len2)) {
ret = -1;
goto cleanup;
}
--
2.54.0.rc1.174.gd833f386ac5.dirty
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox