* [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3
@ 2026-10-23 7:25 Christian Couder
2026-10-23 7:25 ` [PATCH v2 01/13] loose: allow looking up in object map with source Christian Couder
` (14 more replies)
0 siblings, 15 replies; 17+ messages in thread
From: Christian Couder @ 2026-10-23 7:25 UTC (permalink / raw)
To: git
Cc: Junio C Hamano, brian m . carlson, Patrick Steinhardt,
Elijah Newren, Jeff King, Johannes Schindelin, Christian Couder
This series is the next part of brian m. carlson's SHA-1/SHA-256
interoperability work.
It adds support for reading and writing pack index v3, which stores
object IDs in both the main and the compatibility hash
algorithms. Interoperability needs it so that packed objects can be
mapped between algorithms, the same way the loose object map already
does for loose objects.
## Base
This MR targets `master` at 47fdf2208c (The 7th batch, 2026-10-19).
## Origin of the patches
Patches 1-9 and 11-12 are the first 11 commits of brian's
`sha256-interop-part-2` branch (https://github.com/bk2204/git.git,
b175380036), which brian suggested for upstreaming [1].
Patch 10 is a new patch I inserted to validate pack index v3
tables.
Patch 13, which copies pack index v3 documentation, comes from his
full `sha256-interop` branch.
All patches except patch 10 are authored by brian.
[1]: https://lore.kernel.org/git/asV1oB_avuEbgRVe@fruit.crustytoothpaste.net/
## Previous work and naming of this series
Previously, brian sent the following to the mailing list:
- A patch series called `bc/sha1-256-interop-01` merged at 98401c10fc
(Merge branch 'bc/sha1-256-interop-01', 2025-10-22).
- A patch series called `bc/sha1-256-interop-02` merged at 8194f1795b
(Merge branch 'bc/sha1-256-interop-02', 2026-03-12).
So this series is called "part 3" even though it comes mostly from
brian's `sha256-interop-part-2` branch.
## Overview of the patches
- Patches 1-5 add `odb_map_oid()` and `odb_write_oid_map()`, backed
by two new `struct odb_source` callbacks, `map_oid()` and
`write_oid_map()`, which the "files", "loose" and "packed" backends
implement.
- Patches 6-12 implement pack index v3 writing (pack-write,
index-pack) and reading (packfile), and use it to look up compat
object IDs in packs.
- Patch 13 copies the pack index v3 format documentation into
`gitformat-pack(5)`.
## Changes compared to brian's branch
My `Signed-off-by: ...` was added to all the patches. AI assistance
(mostly Claude Opus 5.5) was used.
- Patches 1 and 4 are adapted to `ps/odb-files-alternates`. There is
now a single `odb->source`, so `odb_map_oid()` and
`odb_write_oid_map()` just dispatch to it. The files backend checks
each of its object directories in turn (main directory first, then
alternates) and writes mappings into the main directory, like it
does for objects. The `read_alternates` callbacks are dropped, as
the topic removed them.
- In patch 4, the commit message was improved to describe how the
"files" backend looks up and writes mappings.
- In patch 4, the `odb/source.h` comments about the flags of the
`map_oid()` and `write_oid_map()` callbacks have been fixed. They
wrongly said the flags were unused.
- In patch 6, the `QSORT()` that was added to
`builtin/pack-objects.c` is dropped. It sorted `to_pack.nr_objects`
entries while only the first `nr_written` were set, and it was
redundant anyway, as `write_idx_file()` always sorts the array by
object ID before returning. The comment of `write_idx_file()` and
the commit message now say so.
- In patch 6, when asked to reject duplicate objects (`--strict`),
the v3 writer now looks for them in the arrays sorted by object ID,
where duplicates are adjacent: it compares main object IDs in the
array sorted by main object ID, and compatibility object IDs in the
array sorted by compatibility object ID. (Previously, the main
object IDs were only compared between entries that are next to each
other in pack order, so without a compatibility algorithm, a pack
containing objects A, B, A, B was accepted. With a compatibility
algorithm, the check over the array sorted by compatibility object
ID compared main object IDs, so it did not detect two different
objects with the same compatibility object ID.)
- In patch 7, `index-pack` no longer passes `OBJ_MAP_WRITE` when
recording the compat object IDs it computes, so they stay in memory
as the commit message and the code comment say, like in brian's
full `sha256-interop` branch.
- In patches 7 and 8, the compat hash contexts are initialized with
`git_hash_init()` instead of the raw `init_fn()` callback, as since
9e396aa553 (hash: check ctx->active flag in all wrapper functions,
2026-07-07), `git_hash_update()` refuses contexts that were not
initialized that way. Without this,
`git index-pack --index-version=3` and adding a large file both hit
a `BUG()` in a repository with a compatibility algorithm.
- Patch 7 adds a test to t5308 to check that `index-pack --strict`
rejects duplicate objects when writing a v3 index.
- Patches 7, 8, 11 and 12 now also add tests to t1016 for the
problems fixed in them, which were not covered before. They use a
new small `create_compat_repo()` helper introduced in patch 7.
- In patch 9, a few fixes have been implemented:
- `bsearch_pack_v3()` now reports an object as not found, instead of
dividing by zero, when looking it up in an algorithm for which a
v3 index has no data.
- The `munmap()` calls on errors in `load_idx()` are removed, as the
caller already unmaps the file, like for older index versions.
- The main algorithm is retrieved from the hash size instead of the
repository, so that it works without a repository, for example in
the fuzzer.
- Patch 10 is new. It validates pack index v3 tables, mostly in
`load_idx()`:
- The header length, the trailer offset, the format entries and the
position of all tables are checked against the file size.
- Duplicate formats and invalid short name lengths are rejected.
- When building the reverse index from a v3 index, the 8-byte
offsets are now checked to lie within the index.
- The positions read from the order maps are also checked against
the number of objects when they are used: an invalid one now makes
Git die() with a "corrupt index?" error, or makes a lookup report
the object as not found, instead of reading outside the index. A
test for this is added to t5313.
- In patch 11, mapping now skips packs that don't have a v3 index,
and `nth_packed_object_id_algop()` fails instead of returning bytes
from the wrong table when an index cannot provide the requested
mapping. Previously, in a compat repo, a packed object whose
mapping was not in the loose object map got a wrong compat object
ID.
- In patch 12, looking up objects by an abbreviated compatibility
object ID now reads the object IDs, and the offsets, at the
positions of the table sorted by that algorithm, through
`nth_packed_object_id_algop()` and a new
`nth_packed_object_offset_algop()`. (Previously the main object ID
at that position was compared against the compat prefix, so such
lookups returned wrong or missing results.)
- In patch 12, the compatibility algorithm is also skipped for packs
with an index older than v3, which only store main object IDs. So
when looking up a prefix in any algorithm, the callback no longer
fires twice for the same object in such packs.
- Patch 13 now says that a v3 index has 1 or 2 object formats (both
the reader and the writer accept a single one), and it points to
the hash function transition document for the "Loose objects and
unreachable objects" section, which this document doesn't have.
Except that I added my `Signed-off-by: ...` to them, patches 2, 3 and
5 are unchanged.
## Things not yet implemented
- The test suite can't run with `GIT_TEST_DEFAULT_HASH=sha256:sha1`
yet, because accepting `<algo>:<compat>` in `GIT_DEFAULT_HASH`
comes later in brian's series.
- Index v3 is not yet automatically selected in compat repos, and
`show-index` can't read v3 yet. Both come later in brian's series.
- In a compat repo, `git index-pack` can only compute the compat
object ID of a commit, tree or tag if the objects it references
already have a mapping. As packs usually list commits before their
trees and parents, indexing a pack that brings new objects, for
example from a fetch, fails with "unable to map tree … in commit
object". brian's full `sha256-interop` branch handles this in later
commits, starting with "builtin/index-pack: map straggling
objects", by converting such objects after the whole pack has been
read, in dependency order.
- Resolving an abbreviated compatibility object ID only works with
commands like `git cat-file` or
`git rev-parse --output-object-format=<compat>`, not with a plain
`git rev-parse`, and the packed `find_abbrev_len()` callback
doesn't handle compatibility object IDs yet.
- A multi-pack-index only stores main object IDs, so abbreviated
compatibility object IDs of objects in packs covered by a
multi-pack-index are not found. The t1016 test for abbreviated
compatibility object IDs therefore disables
`GIT_TEST_MULTI_PACK_INDEX`.
- In the "files" backend, `odb_map_oid()` doesn't re-read
info/alternates before retrying. `read_object_info()` doesn't
either on master, but Patrick's "PATCH 0/4 odb/files: reload
alternates on retry again" series [2] restores that for
it. `odb_map_oid()` should follow the same logic once that series
is merged. I have started working on this.
- No protocol support yet.
[2] https://lore.kernel.org/git/20261020-pks-odb-files-reload-alternates-v1-0-d9e156323836@pks.im/
## Changes compared to v1
Thanks to Junio for his comments on this series.
The only change compared to v1 is that I added my `Signed-off-by: ...`
to patches 2, 3 and 5 (that are otherwise unchanged compared to
brian's original patches).
## Range-diff against brian's original work
1: 4e1f77f06a ! 1: 1f24f793f0 loose: allow looking up in object map with source
@@ Commit message
future.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
+ Signed-off-by: Christian Couder <christian.couder@gmail.com>
## loose.c ##
@@ loose.c: int repo_add_loose_object_map(struct odb_source_loose *loose,
@@ loose.c: int repo_add_loose_object_map(struct odb_source_loose *loose,
const struct git_hash_algo *to,
struct object_id *dest)
{
- struct odb_source *source;
+ struct odb_source_files *files = odb_source_files_downcast(repo->objects->source);
- kh_oid_map_t *map;
- khiter_t pos;
- for (source = repo->objects->sources; source; source = source->next) {
- struct odb_source_files *files = odb_source_files_downcast(source);
-- struct loose_object_map *loose_map = files->loose->map;
+ for (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {
+- struct loose_object_map *loose_map = dir->loose->map;
- if (!loose_map)
+ int ret;
+
-+ if (!files->loose)
++ if (!dir->loose)
continue;
- map = (to == repo->compat_hash_algo) ?
- loose_map->to_compat :
@@ loose.c: int repo_add_loose_object_map(struct odb_source_loose *loose,
- oidcpy(dest, kh_value(map, pos));
- return 0;
- }
-+ ret = source_loose_object_map_oid(files->loose, src, to, dest);
++ ret = source_loose_object_map_oid(dir->loose, src, to, dest);
+ if (ret != -1)
+ return ret;
}
2: b36496537a ! 2: 042aed1783 loose: rename repo_add_loose_object_map
@@ Commit message
database abstraction in a future commit.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
+ Signed-off-by: Christian Couder <christian.couder@gmail.com>
## loose.c ##
@@ loose.c: static int write_one_object(struct odb_source_loose *loose,
3: cb123eb81b ! 3: 9da3b460cb loose: add a flag to control whether we write mapped OIDs
@@ Commit message
so pick 1 << 6 as the write bit to avoid conflicts.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
+ Signed-off-by: Christian Couder <christian.couder@gmail.com>
## loose.c ##
@@ loose.c: static int write_one_object(struct odb_source_loose *loose,
4: 4b74282d13 ! 4: 45fe8d6076 odb: implement interoperability functions
@@ Metadata
## Commit message ##
odb: implement interoperability functions
- An object database running with a compatibility algorithm needs a way to
- map object IDs from one algorithm to another and to write mappings into
- the repository. Add these functions and implement them for our loose
- and packed backends so that we can use them in the future.
+ An object database running with a compatibility algorithm needs a way
+ to map object IDs from one algorithm to another and to write mappings
+ into the repository. Add these functions and implement them for the
+ "files" backend and for the "loose" and "packed" sources it is made of,
+ so that we can use them in the future.
- Note that the packed backends don't write any information, since that
+ Like when reading objects, the "files" backend looks up mappings in all
+ of its object directories. It starts with the main one and continues
+ with alternates, which is what the loose object map code already does.
+
+ If a mapping is not found in a directory, the loose object map of that
+ directory is reloaded and checked again, as another process may have
+ written new mappings in the meantime. New mappings are only written
+ into the main object directory, which is also where new objects are
+ written.
+
+ Note that the "packed" sources don't write any information, since that
will be written when we index the pack, and they don't currently read
any information, since we don't yet have support for pack index v3,
which we'll need to read this properly. This functionality will come
in a future commit.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
+ Signed-off-by: Christian Couder <christian.couder@gmail.com>
## odb.c ##
@@ odb.c: int odb_count_objects(struct object_database *odb,
@@ odb.c: int odb_count_objects(struct object_database *odb,
+ struct object_id *dest,
+ enum odb_map_oid_flags flags)
+{
-+ struct odb_source *source;
-+ int ret;
-+
-+ odb_prepare_alternates(odb, NULL);
-+ for (source = odb->sources; source; source = source->next) {
-+ ret = odb_source_map_oid(source, src, to, dest, flags);
-+ if (ret != -1)
-+ return ret;
-+ }
-+ return -1;
++ return odb_source_map_oid(odb->source, src, to, dest, flags);
+}
+
+int odb_write_oid_map(struct object_database *odb,
@@ odb.c: int odb_count_objects(struct object_database *odb,
+ const struct object_id *compat_oid,
+ enum odb_write_oid_map_flags flags)
+{
-+ struct odb_source *source;
-+
-+ odb_prepare_alternates(odb, NULL);
-+ for (source = odb->sources; source; source = source->next)
-+ if (!odb_source_write_oid_map(source, oid, compat_oid, flags))
-+ return 0;
-+ return -1;
++ return odb_source_write_oid_map(odb->source, oid, compat_oid, flags);
+}
++
/*
* Return the slot of the most-significant bit set in "val". There are various
* ways to do this quickly with fls() or __builtin_clzl(), but speed is
@@ odb/source-files.c
#include "object-file.h"
#include "odb.h"
#include "odb/source.h"
-@@ odb/source-files.c: static int odb_source_files_fsck(struct odb_source *source,
- return ret;
+@@ odb/source-files.c: struct odb_files_dir *odb_source_files_find_dir(struct object_database *odb, con
+ return dir;
}
+static int odb_source_files_map_oid(struct odb_source *source,
@@ odb/source-files.c: static int odb_source_files_fsck(struct odb_source *source,
+ enum odb_map_oid_flags flags)
+{
+ struct odb_source_files *files = odb_source_files_downcast(source);
-+ int ret;
-+ ret = odb_source_map_oid(&files->loose->base, src, to, dest, flags);
-+ if (ret != -1)
-+ return ret;
-+ ret = odb_source_map_oid(&files->packed->base, src, to, dest, flags);
-+ if (ret != -1)
-+ return ret;
-+ return odb_source_map_oid(&files->loose->base, src, to, dest, flags | ODB_MAP_OID_RELOAD);
++
++ for (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {
++ int ret;
++
++ ret = odb_source_map_oid(&dir->loose->base, src, to, dest, flags);
++ if (ret != -1)
++ return ret;
++ ret = odb_source_map_oid(&dir->packed->base, src, to, dest, flags);
++ if (ret != -1)
++ return ret;
++ ret = odb_source_map_oid(&dir->loose->base, src, to, dest,
++ flags | ODB_MAP_OID_RELOAD);
++ if (ret != -1)
++ return ret;
++ }
++
++ return -1;
+}
+
+static inline int odb_source_files_write_oid_map(struct odb_source *source,
@@ odb/source-files.c: static int odb_source_files_fsck(struct odb_source *source,
+ enum odb_write_oid_map_flags flags)
+{
+ struct odb_source_files *files = odb_source_files_downcast(source);
-+ return odb_source_write_oid_map(&files->loose->base, oid, compat_oid, flags);
++ return odb_source_write_oid_map(&files->dirs->loose->base, oid, compat_oid, flags);
+}
+
struct odb_source_files *odb_source_files_new(struct object_database *odb,
- const char *path,
- bool local)
+ enum odb_new_flags flags)
+ {
@@ odb/source-files.c: struct odb_source_files *odb_source_files_new(struct object_database *odb,
+ files->base.write_object = odb_source_files_write_object;
files->base.write_object_stream = odb_source_files_write_object_stream;
files->base.begin_transaction = odb_source_files_begin_transaction;
- files->base.read_alternates = odb_source_files_read_alternates;
+ files->base.map_oid = odb_source_files_map_oid;
+ files->base.write_oid_map = odb_source_files_write_oid_map;
files->base.optimize = odb_source_files_optimize;
@@ odb/source-loose.c: static void odb_source_loose_reparent(const char *old_cwd,
+ * let's reload the map to see if the object has appeared.
+ */
+ if (flags & ODB_MAP_OID_RELOAD)
-+ repo_read_loose_object_map(source->odb->repo);
++ loose_object_map_load(loose);
+ return source_loose_object_map_oid(loose, src, to, dest);
+}
+
@@ odb/source-loose.c: static void odb_source_loose_reparent(const char *old_cwd,
{
struct odb_source_loose *loose = odb_source_loose_downcast(source);
@@ odb/source-loose.c: struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
+ loose->base.write_object = odb_source_loose_write_object;
loose->base.write_object_stream = odb_source_loose_write_object_stream;
loose->base.begin_transaction = odb_source_loose_begin_transaction;
- loose->base.read_alternates = odb_source_loose_read_alternates;
+ loose->base.map_oid = odb_source_loose_map_oid;
+ loose->base.write_oid_map = odb_source_loose_write_oid_map;
@@ odb/source-packed.c: static void odb_source_packed_close(struct odb_source *sour
{
struct odb_source_packed *packed = odb_source_packed_downcast(source);
@@ odb/source-packed.c: struct odb_source_packed *odb_source_packed_new(struct object_database *odb,
+ packed->base.write_object = odb_source_packed_write_object;
packed->base.write_object_stream = odb_source_packed_write_object_stream;
packed->base.begin_transaction = odb_source_packed_begin_transaction;
- packed->base.read_alternates = odb_source_packed_read_alternates;
+ packed->base.map_oid = odb_source_packed_map_oid;
+ packed->base.write_oid_map = odb_source_packed_write_oid_map;
@@ odb/source.h: struct odb_source {
+ /*
+ * This callback is expected to provide the mapping of an object from
+ * the algorithm specified in src to the algorithm specified in to,
-+ * leaving the result in dest. flags is required, but so far unused; it
-+ * should be set to 0.
++ * leaving the result in dest. flags is a combination of `enum
++ * odb_map_oid_flags` values: with ODB_MAP_OID_RELOAD, the source should
++ * reload its mapping data before the lookup, as another process may
++ * have written new mappings in the meantime.
+ *
+ * The algo field in src must point to a valid algorithm (specifically,
+ * it must not be zero). to must likewise point to a valid algorithm;
@@ odb/source.h: struct odb_source {
+
+ /*
+ * This callback is expected to write a mapping of an object between oid
-+ * and compat_oid. flags is required, but so far unused; it should be
-+ * set to 0.
++ * and compat_oid. flags is a combination of `enum
++ * odb_write_oid_map_flags` values: with ODB_WRITE_OID_MAP_WRITE, the
++ * mapping should be persisted to disk; otherwise it may only be kept in
++ * memory.
+ *
+ * The algo fields in oid and compat_oid must point to valid algorithms
+ * (specifically, they must not be zero).
5: c5781631c6 ! 5: 69dc3ffea6 object-file-convert: implement mapping in terms of object DB
@@ Commit message
issues.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
+ Signed-off-by: Christian Couder <christian.couder@gmail.com>
## object-file-convert.c ##
@@
6: 885edd8636 ! 6: c2123917c7 pack-write: add support for pack index v3
@@ Commit message
doing so will break the tests. In a future commit, once we can read it,
we'll enable it by default when a compatibility algorithm is enabled.
- Since we're not sorting the array unconditionally in write_idx_file, be
- sure to sort it where necessary.
+ Index v3 needs the objects both in pack order and sorted by object ID,
+ so when writing a v3 index, sort copies of the array instead of the
+ array itself. Callers, like the bitmap code in pack-objects, rely on
+ the array being sorted by object ID when write_idx_file() returns,
+ though, so sort it before returning in that case too.
- Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
+ When asked to reject duplicate objects using `--strict`, look for them
+ in the arrays sorted by object ID, where duplicates are adjacent.
- ## builtin/pack-objects.c ##
-@@
- */
- static struct packing_data to_pack;
-
-+static int oid_compare(const void *_a, const void *_b)
-+{
-+ struct pack_idx_entry *a = *(struct pack_idx_entry **)_a;
-+ struct pack_idx_entry *b = *(struct pack_idx_entry **)_b;
-+ return oidcmp(&a->oid, &b->oid);
-+}
-+
- static inline struct object_entry *oe_delta(
- const struct packing_data *pack,
- const struct object_entry *e)
-@@ builtin/pack-objects.c: static void write_pack_file(void)
- if (write_bitmap_index) {
- size_t tmpname_len = tmpname.len;
-
-+ QSORT(written_list, to_pack.nr_objects, oid_compare);
-+
- strbuf_addstr(&tmpname, "bitmap");
- stop_progress(&progress_state);
-
+ Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
+ Signed-off-by: Christian Couder <christian.couder@gmail.com>
## pack-write.c ##
@@ pack-write.c: void reset_pack_idx_option(struct pack_idx_option *opts)
@@ pack-write.c: static int need_large_offset(off_t offset, const struct pack_idx_o
- * The *sha1 contains the pack content SHA1 hash.
- * The objects array passed in will be sorted by SHA1 on exit.
+ * The *hash contains the pack content hash.
-+ * The objects array passed in should be in pack order and may be sorted (or
-+ * not) on return.
++ * The objects array passed in should be in pack order and will be sorted by
++ * object ID on return.
*/
const char *write_idx_file(struct repository *repo,
const char *index_name, struct pack_idx_entry **objects,
@@ pack-write.c: const char *write_idx_file(struct repository *repo,
+ for (i = 0; i < nr_objects; i++) {
+ struct pack_idx_entry *obj = *list++;
+ hashwrite(f, obj->oid.hash, short_name_len[0]);
++ if ((opts->flags & WRITE_IDX_STRICT) &&
++ (i && oideq(&list[-2]->oid, &obj->oid)))
++ die("The same object %s appears twice in the pack",
++ oid_to_hex(&obj->oid));
+ }
+ } else {
+ /*
@@ pack-write.c: const char *write_idx_file(struct repository *repo,
for (i = 0; i < nr_objects; i++) {
struct pack_idx_entry *obj = *list++;
if (index_version < 2)
-@@ pack-write.c: const char *write_idx_file(struct repository *repo,
+ hashwrite_be32(f, obj->offset);
+ hashwrite(f, obj->oid.hash, repo->hash_algo->rawsz);
+- if ((opts->flags & WRITE_IDX_STRICT) &&
++ if ((opts->flags & WRITE_IDX_STRICT) && index_version < 3 &&
+ (i && oideq(&list[-2]->oid, &obj->oid)))
+ die("The same object %s appears twice in the pack",
oid_to_hex(&obj->oid));
}
@@ pack-write.c: const char *write_idx_file(struct repository *repo,
+ struct pack_idx_entry *obj = *list++;
+ hashwrite(f, obj->compat_oid.hash, short_name_len[1]);
+ if ((opts->flags & WRITE_IDX_STRICT) &&
-+ (i && oideq(&list[-2]->oid, &obj->oid)))
++ (i && oideq(&list[-2]->compat_oid, &obj->compat_oid)))
+ die("The same object %s appears twice in the pack",
+ oid_to_hex(&obj->compat_oid));
+ }
@@ pack-write.c: const char *write_idx_file(struct repository *repo,
+ for (i = 0; i < nr_objects; i++) {
+ struct pack_idx_entry *obj = *list++;
+ hashwrite(f, obj->compat_oid.hash, compat->rawsz);
-+ if ((opts->flags & WRITE_IDX_STRICT) &&
-+ (i && oideq(&list[-2]->oid, &obj->oid)))
-+ die("The same object %s appears twice in the pack",
-+ oid_to_hex(&obj->compat_oid));
+ }
+
+ /* write the sorted-to-pack index */
7: fe70d52c3a ! 7: 8ced7b7a0b builtin: populate pack index v3 fields when indexing
@@ Commit message
and release the mutex between the conversion and writing to the map
because hashing may be slow.
+ Add a test in t1016 to check that index-pack can write a v3 index in a
+ repository with a compatibility algorithm, and a test that it rejects
+ packs with duplicate objects when writing a v3 index with `--strict`.
+
+ Also add a create_compat_repo() helper in t1016 to refactor common
+ code.
+
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
+ Signed-off-by: Christian Couder <christian.couder@gmail.com>
## builtin/index-pack.c ##
@@
@@ builtin/index-pack.c: static void *unpack_entry_data(off_t offset, size_t size,
git_hash_update(&c, hdr, hdrlen);
- } else
+ if (compat && compat_oid) {
-+ compat->init_fn(&c_compat);
++ git_hash_init(&c_compat, compat);
+ if (type == OBJ_BLOB)
-+ compat->update_fn(&c_compat, hdr, hdrlen);
++ git_hash_update(&c_compat, hdr, hdrlen);
+ }
+ } else {
oid = NULL;
@@ builtin/index-pack.c: static void *unpack_entry_data(off_t offset, size_t size,
+ }
+ git_hash_final_oid(compat_oid, &c_compat);
+ repo_lock();
-+ odb_write_oid_map(the_repository->objects, oid, compat_oid, OBJ_MAP_WRITE);
++ odb_write_oid_map(the_repository->objects, oid, compat_oid, 0);
+ repo_unlock();
+ }
return buf == fixed_buf ? NULL : buf;
@@ builtin/index-pack.c: static struct base_data *resolve_delta(struct object_entry
+ repo_lock();
+ odb_write_oid_map(the_repository->objects,
+ &delta_obj->idx.oid,
-+ &delta_obj->idx.compat_oid,
-+ OBJ_MAP_WRITE);
++ &delta_obj->idx.compat_oid, 0);
+ repo_unlock();
+ strbuf_release(&outbuf);
+ }
@@ builtin/pack-objects.c: static int option_parse_index_version(const struct optio
die(_("unsupported index version %s"), val);
if (*c == ',' && c[1])
popts->off32_limit = strtoul(c+1, &c, 0);
+
+ ## t/t1016-compatObjectFormat.sh ##
+@@ t/t1016-compatObjectFormat.sh: compat_hash () {
+ esac
+ }
+
++create_compat_repo () {
++ local hash="$1"
++ local repo="$2"
++
++ git init --object-format="$hash" "$repo" &&
++ git -C "$repo" config set core.repositoryformatversion 1 &&
++ git -C "$repo" config set extensions.compatObjectFormat $(compat_hash "$hash")
++}
++
+ hello_oid () {
+ case "$1" in
+ "sha1")
+@@ t/t1016-compatObjectFormat.sh: do
+ for repo in alt borrow
+ do
+ test_when_finished "rm -rf $repo" &&
+- git init --object-format=$hash $repo &&
+- git -C $repo config set core.repositoryformatversion 1 &&
+- git -C $repo config set extensions.compatObjectFormat $(compat_hash $hash) || exit 1
++ create_compat_repo $hash $repo || exit 1
+ done &&
+
+ git -C alt commit --allow-empty --message A &&
+@@ t/t1016-compatObjectFormat.sh: do
+ git -C borrow rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
+ test_cmp expect actual
+ '
++
++ test_expect_success 'index-pack writes pack index v3' '
++ test_when_finished "rm -rf v3-repo v3.idx" &&
++ create_compat_repo $hash v3-repo &&
++ echo content >v3-repo/file &&
++ git -C v3-repo add file &&
++ git -C v3-repo commit --message A &&
++ git -C v3-repo commit --allow-empty --message B &&
++ git -C v3-repo repack -a -d &&
++ pack=$(echo v3-repo/.git/objects/pack/pack-*.pack) &&
++ git -C v3-repo index-pack --index-version=3 \
++ -o "$(pwd)/v3.idx" "$(pwd)/$pack" &&
++ echo "ff 74 4f 63 00 00 00 03 " >expect &&
++ test_copy_bytes 8 <v3.idx | test-tool hexdump >actual &&
++ test_cmp expect actual
++ '
+ done
+ cd "$base"
+
+
+ ## t/t5308-pack-detect-duplicates.sh ##
+@@ t/t5308-pack-detect-duplicates.sh: test_expect_success 'index-pack can reject packs with duplicates' '
+ test_expect_code 1 git cat-file -e $LO_SHA1
+ '
+
++test_expect_success 'index-pack can reject packs with duplicates with index v3' '
++ clear_packs &&
++ create_pack dups.pack 2 &&
++ test_must_fail git index-pack --strict --index-version=3 --stdin <dups.pack &&
++ test_expect_code 1 git cat-file -e $LO_SHA1
++'
++
+ test_done
8: cb593fd7dd ! 8: 75e70ed1c1 object-file: handle compatibility algorithm
@@ Commit message
the main algorithm or we will incorrectly store the null OID in the
index we create. Do so here to avoid corrupting the repository.
+ Add a test that adds a large file in a repository with a compatibility
+ algorithm.
+
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
+ Signed-off-by: Christian Couder <christian.couder@gmail.com>
## object-file.c ##
@@ object-file.c: static int hash_stream(struct odb_stream *stream,
@@ object-file.c: static int odb_transaction_files_write_object_stream(struct odb_t
git_hash_init(&ctx, transaction->base.source->odb->repo->hash_algo);
git_hash_update(&ctx, obuf, header_len);
+ if (compat) {
-+ compat->init_fn(&compat_ctx);
++ git_hash_init(&compat_ctx, compat);
+ git_hash_update(&compat_ctx, obuf, header_len);
+ }
@@ object-file.c: static int odb_transaction_files_write_object_stream(struct odb_t
ALLOC_GROW(state->written,
state->nr_written + 1,
state->alloc_written);
+
+ ## t/t1016-compatObjectFormat.sh ##
+@@ t/t1016-compatObjectFormat.sh: do
+ test_copy_bytes 8 <v3.idx | test-tool hexdump >actual &&
+ test_cmp expect actual
+ '
++
++ test_expect_success 'add a large file streamed into a pack' '
++ test_when_finished "rm -rf large-repo" &&
++ create_compat_repo $hash large-repo &&
++ git -C large-repo config set core.bigFileThreshold 1k &&
++ test-tool genrandom large 5000 >large-repo/large &&
++ git -C large-repo add large &&
++ ls large-repo/.git/objects/pack/pack-*.pack &&
++ git -C large-repo hash-object large >expect &&
++ git -C large-repo rev-parse :large >actual &&
++ test_cmp expect actual &&
++ git -C large-repo cat-file blob :large >actual &&
++ test_cmp large-repo/large actual
++ '
+ done
+ cd "$base"
+
9: c401efada9 ! 9: 036a8fa8cc packfile: add support for reading pack index v3
@@ Commit message
larger. We can then use it when finding the offset into the file, since
it will be generic over both v2 and v3 indices.
+ A v3 index may contain data for only one hash algorithm, so just fail if
+ we look up an object ID in an algorithm for which the index has no data.
+
In the pack revindex, we use the order map for the algorithm in question
to look up the proper offset into the pack index, since we're iterating
over objects in the offset table, which is sorted by object ID, not pack
@@ Commit message
formats can't represent the SHA-1 values.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
+ Signed-off-by: Christian Couder <christian.couder@gmail.com>
## builtin/index-pack.c ##
@@ builtin/index-pack.c: static void read_idx_option(struct pack_idx_option *opts, const char *pack_name)
@@ packfile.c: int load_idx(const char *path, const unsigned int hashsz, void *idx_
return error("pack too large for current definition of off_t in %s", path);
p->crc_offset = st_add(8 + 4 * 256, st_mult(nr, hashsz));
+ } else if (version == 3) {
-+ uint32_t hdrlen, num_formats, trailer_offset;
++ uint32_t hdrlen, num_formats, trailer_offset, main_algo;
+ const uint32_t *q = index;
+ int i;
+
+ hdrlen = ntohl(*q++);
-+ if (idx_size < hdrlen) {
-+ munmap(idx_map, idx_size);
++ if (idx_size < hdrlen)
+ return error("v3 index has truncated header");
-+ }
+
+ nr = ntohl(*q++);
+ num_formats = ntohl(*q++);
-+ if (num_formats < 1 || num_formats > ARRAY_SIZE(p->formats)) {
-+ munmap(idx_map, idx_size);
++ if (num_formats < 1 || num_formats > ARRAY_SIZE(p->formats))
+ return error("too many formats for v3 index");
-+ }
+ memset(p->formats, 0, sizeof(p->formats));
+ for (i = 0; i < num_formats; i++) {
+ uint32_t format_id = ntohl(*q++);
+ uint32_t format = hash_algo_by_id(format_id);
+ struct packed_git_format *pfp;
+
-+ if (format == GIT_HASH_UNKNOWN) {
-+ munmap(idx_map, idx_size);
++ if (format == GIT_HASH_UNKNOWN)
+ return error("v3 index with unknown format_id 0x%08x", format_id);
-+ }
+ /*
+ * We store at the value of the constant minus 1 to
+ * avoid having to store a useless entry at offset 0.
@@ packfile.c: int load_idx(const char *path, const unsigned int hashsz, void *idx_
+ if (!i)
+ p->crc_offset = pfp->order_map_offset + nr * 4;
+ }
-+ if (!p->formats[p->repo->hash_algo - hash_algos - 1].data_offset) {
-+ munmap(idx_map, idx_size);
++ main_algo = hash_algo_by_length(hashsz);
++ if (main_algo == GIT_HASH_UNKNOWN ||
++ !p->formats[main_algo - 1].data_offset)
+ return error("v3 index with missing data for this repository's algorithm");
-+ }
+ trailer_offset = get_be64(q);
-+ if (trailer_offset + 2 * hashsz != idx_size) {
-+ munmap(idx_map, idx_size);
++ if (trailer_offset + 2 * hashsz != idx_size)
+ return error("v3 index with bad size\n");
-+ }
}
p->index_version = version;
@@ packfile.c: void *unpack_entry(struct repository *r, struct packed_git *p, off_t
+ const struct packed_git_format *pfp = &p->formats[algo - 1];
+ const unsigned char *data = p->index_data;
+
++ /* This index has no data for the algorithm of the object ID. */
++ if (!pfp->data_offset) {
++ if (result)
++ *result = 0;
++ return 0;
++ }
++
+ return bsearch_hash_v3(oid->hash, data + pfp->full_oid_offset,
+ (const uint32_t *)(data + pfp->order_map_offset),
+ data + pfp->data_offset,
-: ---------- > 10: 5e8aaa8329 packfile: validate pack index v3 tables
10: cbfbb56596 ! 11: 4d575f34ed loose: map object IDs using pack index v3
@@ Commit message
In order to do so, let's add a variant of nth_packed_object_id that
handles an arbitrary hash algorithm.
+ Only pack index v3 stores object IDs in more than one algorithm, so
+ skip packs with an older index when mapping.
+
+ Also make the new variant fail when the index cannot provide the
+ requested object ID, for example because it has no data for one of the
+ algorithms.
+
+ Add a test in t1016 to check that packed objects are mapped using a v3
+ index, but not using a v2 index, when the loose object map has no entry
+ for them.
+
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
+ Signed-off-by: Christian Couder <christian.couder@gmail.com>
## odb/source-packed.c ##
@@ odb/source-packed.c: static void odb_source_packed_close(struct odb_source *source)
@@ odb/source-packed.c: static void odb_source_packed_close(struct odb_source *sour
+
+ if (open_pack_index(p) || !p->num_objects)
+ continue;
++ /* Only v3 indices store object IDs in both algorithms. */
++ if (p->index_version < 3)
++ continue;
+ if (!bsearch_pack(src, p, &off))
+ continue;
-+ return nth_packed_object_id_algop(dest, p, off, src_algo, to);
++ if (nth_packed_object_id_algop(dest, p, off, src_algo, to))
++ continue;
++ return 0;
+ }
return -1;
}
@@ packfile.c: int bsearch_pack(const struct object_id *oid, const struct packed_gi
if (open_pack_index(p))
return -1;
@@ packfile.c: int nth_packed_object_id(struct object_id *oid,
+ }
+ if (n >= p->num_objects)
+ return -1;
++ if (p->index_version < 3 && (src != dest || dest != p->repo->hash_algo))
++ return -1; /* older indices only store main object IDs */
if (p->index_version == 1) {
index += 4 * 256;
oidread(oid, index + st_add(st_mult(hashsz + 4, n), 4),
@@ packfile.c: int nth_packed_object_id(struct object_id *oid,
+ int algo = hash_algo_by_ptr(dest);
const struct packed_git_format *pfp = &p->formats[algo-1];
- oidread(oid, index + pfp->full_oid_offset + (hashsz * n), p->repo->hash_algo);
-+ uint32_t pack_order_idx = nth_packed_object_pack_order_algop(p, n, src);
++ uint32_t pack_order_idx;
++
++ if (!pfp->data_offset ||
++ !p->formats[hash_algo_by_ptr(src) - 1].data_offset)
++ return -1; /* no data for one of the algorithms */
++ pack_order_idx = nth_packed_object_pack_order_algop(p, n, src);
+ oidread(oid, index + st_add(pfp->full_oid_offset, st_mult(hashsz, pack_order_idx)), dest);
}
return 0;
@@ packfile.h: int bsearch_pack(const struct object_id *oid, const struct packed_gi
/*
* Return the offset of the nth object within the specified packfile.
* The index must already be opened.
+
+ ## t/t1016-compatObjectFormat.sh ##
+@@ t/t1016-compatObjectFormat.sh: do
+ git -C large-repo cat-file blob :large >actual &&
+ test_cmp large-repo/large actual
+ '
++
++ test_expect_success 'packed objects are only mapped using pack index v3' '
++ test_when_finished "rm -rf map-repo v3.idx loose-object-idx" &&
++ create_compat_repo $hash map-repo &&
++ echo content >map-repo/file &&
++ git -C map-repo add file &&
++ git -C map-repo commit --message A &&
++ oid=$(git -C map-repo rev-parse HEAD) &&
++ git -C map-repo rev-parse --output-object-format=$(compat_hash $hash) "$oid" >expect &&
++ git -C map-repo repack -a -d &&
++ pack=$(echo map-repo/.git/objects/pack/pack-*.pack) &&
++ git -C map-repo index-pack --index-version=3 \
++ -o "$(pwd)/v3.idx" "$(pwd)/$pack" &&
++ mv map-repo/.git/objects/loose-object-idx loose-object-idx &&
++
++ # A v2 index only has object IDs in the main algorithm, so it
++ # must not provide a mapping.
++ git -C map-repo rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
++ ! test_cmp expect actual &&
++ test $(wc -c <actual) -ne $(($(test_oid --hash=$(compat_hash $hash) hexsz) + 1)) &&
++
++ # A v3 index provides it.
++ rm -f "${pack%.pack}.idx" "${pack%.pack}.rev" &&
++ mv v3.idx "${pack%.pack}.idx" &&
++ git -C map-repo rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
++ test_cmp expect actual
++ '
+ done
+ cd "$base"
+
11: f60e1e2c0d ! 12: ce5de5c8c5 packfile: allow looking up compat object IDs in packs
@@ Commit message
it is easier to understand as two statements. The compiler will
optimize it appropriately.
+ When looking up an object ID, we find positions in the table sorted by
+ that object ID's algorithm. So read the object IDs there for that
+ algorithm with nth_packed_object_id_algop(), and add
+ nth_packed_object_offset_algop() to get their offsets.
+
+ Skip the compatibility algorithm for packs with an index older than v3,
+ as these only store object IDs in the main algorithm.
+
+ Add a test that looks up an abbreviated compatibility object ID of a
+ packed object.
+
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
+ Signed-off-by: Christian Couder <christian.couder@gmail.com>
## odb/source-packed.c ##
@@ odb/source-packed.c: static int for_each_prefixed_object_in_pack(
@@ odb/source-packed.c: static int for_each_prefixed_object_in_pack(
+ (!p->repo->compat_hash_algo ||
+ algop != p->repo->compat_hash_algo))
+ continue;
-+
++ if (p->index_version < 3 && algop != p->repo->hash_algo)
++ continue;
+
+ oidcpy(&wanted, opts->prefix);
+ wanted.algo = algo;
@@ odb/source-packed.c: static int for_each_prefixed_object_in_pack(
+ for (i = first; i < num; i++) {
+ struct object_id oid;
+
-+ nth_packed_object_id(&oid, p, i);
-+ if (!match_hash(len, opts->prefix->hash, oid.hash))
++ if (nth_packed_object_id_algop(&oid, p, i, algop, algop) ||
++ !match_hash(len, opts->prefix->hash, oid.hash))
+ break;
+
+ if (data->request) {
+ struct object_info oi = *data->request;
-+ off_t offset = nth_packed_object_offset(p, i);
++ off_t offset = nth_packed_object_offset_algop(p, i, algop);
+
+ ret = packed_object_info(source, p, offset, &oi);
+ if (ret)
@@ odb/source-packed.c: static int for_each_prefixed_object_in_pack(
}
}
+
+ ## packfile.c ##
+@@ packfile.c: void check_pack_index_ptr(const struct packed_git *p, const void *vptr)
+ }
+
+ off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)
++{
++ return nth_packed_object_offset_algop(p, n, p->repo->hash_algo);
++}
++
++off_t nth_packed_object_offset_algop(const struct packed_git *p, uint32_t n,
++ const struct git_hash_algo *algop)
+ {
+ const unsigned char *index = p->index_data;
+ const unsigned int hashsz = p->repo->hash_algo->rawsz;
++
++ if (p->index_version < 3 && algop != p->repo->hash_algo)
++ BUG("pack index v%d only stores main object IDs",
++ p->index_version);
++
+ if (p->index_version == 1) {
+ index += 4 * 256;
+ return ntohl(*((uint32_t *)(index + st_mult(hashsz + 4, n))));
+ } else {
+ uint32_t off;
+ if (p->index_version == 3)
+- n = nth_packed_object_pack_order_algop(p, n, p->repo->hash_algo);
++ n = nth_packed_object_pack_order_algop(p, n, algop);
+ index += st_add(p->crc_offset, st_mult(p->num_objects, 4));
+ off = ntohl(*((uint32_t *)(index + st_mult(4, n))));
+ if (!(off & 0x80000000))
+
+ ## packfile.h ##
+@@ packfile.h: int nth_packed_object_id_algop(struct object_id *, struct packed_git *,
+ */
+ off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);
+
++/*
++ * The same as nth_packed_object_offset, but the nth object is counted in
++ * algop. For indices older than v3, algop must be the main algorithm.
++ */
++off_t nth_packed_object_offset_algop(const struct packed_git *, uint32_t n,
++ const struct git_hash_algo *algop);
++
+ /*
+ * Return the index into pack order of the nth object within the specified
+ * packfile. The index must already be opened and it must be a v3 index.
+
+ ## t/t1016-compatObjectFormat.sh ##
+@@ t/t1016-compatObjectFormat.sh: do
+ git -C map-repo rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
+ test_cmp expect actual
+ '
++
++ test_expect_success 'abbreviated compat object IDs are found using pack index v3' '
++ test_when_finished "rm -rf abbrev-repo v3.idx" &&
++ create_compat_repo $hash abbrev-repo &&
++ echo content >abbrev-repo/file &&
++ git -C abbrev-repo add file &&
++ git -C abbrev-repo commit --message A &&
++ compat_oid=$(git -C abbrev-repo rev-parse --output-object-format=$(compat_hash $hash) HEAD) &&
++ abbrev=$(echo $compat_oid | cut -c 1-12) &&
++
++ # A multi-pack-index only stores object IDs in the main
++ # algorithm.
++ GIT_TEST_MULTI_PACK_INDEX=0 git -C abbrev-repo repack -a -d &&
++ pack=$(echo abbrev-repo/.git/objects/pack/pack-*.pack) &&
++ git -C abbrev-repo index-pack --index-version=3 \
++ -o "$(pwd)/v3.idx" "$(pwd)/$pack" &&
++ rm abbrev-repo/.git/objects/loose-object-idx &&
++
++ # A v2 index only has object IDs in the main algorithm.
++ test_must_fail git -C abbrev-repo cat-file -t $abbrev &&
++
++ rm -f "${pack%.pack}.idx" "${pack%.pack}.rev" &&
++ mv v3.idx "${pack%.pack}.idx" &&
++ echo commit >expect &&
++ git -C abbrev-repo cat-file -t $abbrev >actual &&
++ test_cmp expect actual &&
++ echo $compat_oid >expect &&
++ git -C abbrev-repo rev-parse --output-object-format=$(compat_hash $hash) $abbrev >actual &&
++ test_cmp expect actual
++ '
+ done
+ cd "$base"
+
12: 6afaf2ff9e ! 13: 50f515b9aa doc: move pack index v3 documentation
@@ Commit message
compatibility.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
+ Signed-off-by: Christian Couder <christian.couder@gmail.com>
## Documentation/gitformat-pack.adoc ##
@@ Documentation/gitformat-pack.adoc: Pack file entry: <+
@@ Documentation/gitformat-pack.adoc: Pack file entry: <+
+ * 4-byte length of the header section, including the signature and
+ version number
+ * 4-byte number of objects contained in the pack
-+ * 4-byte number of object formats in this pack index: 2
++ * 4-byte number of object formats in this pack index: 1 or 2
+ * For each object format:
+ ** 4-byte format identifier (e.g., 'sha1' for SHA-1)
+ ** 4-byte length in bytes of shortened object names. This is the
@@ Documentation/gitformat-pack.adoc: Pack file entry: <+
+ * 8-byte offset to the trailer from the beginning of this file.
+ * Zero or more additional key/value pairs (4-byte key, 4-byte
+ value). Only one key is supported: 'PSRC'. See the "Loose objects
-+ and unreachable objects" section for supported values and how this
-+ is used. All other keys are reserved. Readers must ignore
++ and unreachable objects" section of the hash function transition
++ document (`Documentation/technical/hash-function-transition.adoc`)
++ for supported values and how this is used. All other keys are reserved. Readers must ignore
+ unrecognized keys.
+ - Zero or more NUL bytes. This can optionally be used to improve the
+ alignment of the full object name table below.
Christian Couder (1):
packfile: validate pack index v3 tables
brian m. carlson (12):
loose: allow looking up in object map with source
loose: rename repo_add_loose_object_map
loose: add a flag to control whether we write mapped OIDs
odb: implement interoperability functions
object-file-convert: implement mapping in terms of object DB
pack-write: add support for pack index v3
builtin: populate pack index v3 fields when indexing
object-file: handle compatibility algorithm
packfile: add support for reading pack index v3
loose: map object IDs using pack index v3
packfile: allow looking up compat object IDs in packs
doc: move pack index v3 documentation
Documentation/gitformat-pack.adoc | 75 +++++++++
builtin/index-pack.c | 107 +++++++++++-
builtin/pack-objects.c | 8 +-
hash-lookup.c | 50 +++++-
hash-lookup.h | 19 ++-
loose.c | 50 ++++--
loose.h | 15 +-
object-file-convert.c | 14 +-
object-file.c | 18 +-
odb.c | 17 ++
odb.h | 24 +++
odb/source-files.c | 38 +++++
odb/source-loose.c | 31 +++-
odb/source-packed.c | 124 ++++++++++----
odb/source.h | 65 ++++++++
pack-check.c | 4 +-
pack-revindex.c | 24 ++-
pack-write.c | 263 ++++++++++++++++++++++++++----
pack.h | 2 +
packfile.c | 185 +++++++++++++++++++--
packfile.h | 33 ++++
t/t1016-compatObjectFormat.sh | 100 +++++++++++-
t/t5302-pack-index.sh | 71 ++++----
t/t5308-pack-detect-duplicates.sh | 7 +
t/t5313-pack-bounds-checks.sh | 17 ++
25 files changed, 1193 insertions(+), 168 deletions(-)
--
2.56.0.215.g47fdf2208c
^ permalink raw reply [flat|nested] 17+ messages in thread
* [PATCH v2 01/13] loose: allow looking up in object map with source
2026-10-23 7:25 [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Christian Couder
@ 2026-10-23 7:25 ` Christian Couder
2026-10-23 7:25 ` [PATCH v2 02/13] loose: rename repo_add_loose_object_map Christian Couder
` (13 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Christian Couder @ 2026-10-23 7:25 UTC (permalink / raw)
To: git
Cc: Junio C Hamano, brian m . carlson, Patrick Steinhardt,
Elijah Newren, Jeff King, Johannes Schindelin, Christian Couder
From: "brian m. carlson" <sandals@crustytoothpaste.net>
In preparation for wiring up object mapping into the ODB abstraction,
add a function which can map object IDs based on a struct
odb_source_loose and use it in the existing function. This preserves
the same behavior, but allows us to make use of it elsewhere in the
future.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Christian Couder <christian.couder@gmail.com>
---
loose.c | 41 +++++++++++++++++++++++++++++------------
loose.h | 5 +++++
2 files changed, 34 insertions(+), 12 deletions(-)
diff --git a/loose.c b/loose.c
index d6a2bb5ba3..fe8f540979 100644
--- a/loose.c
+++ b/loose.c
@@ -225,27 +225,44 @@ int repo_add_loose_object_map(struct odb_source_loose *loose,
return 0;
}
+int source_loose_object_map_oid(struct odb_source_loose *loose,
+ const struct object_id *src,
+ const struct git_hash_algo *to,
+ struct object_id *dest)
+{
+ kh_oid_map_t *map;
+ khiter_t pos;
+ struct loose_object_map *loose_map = loose->map;
+ struct repository *repo = loose->base.odb->repo;
+
+ if (!loose_map)
+ return -1;
+ map = (to == repo->compat_hash_algo) ?
+ loose_map->to_compat :
+ loose_map->to_storage;
+ pos = kh_get_oid_map(map, *src);
+ if (pos < kh_end(map)) {
+ oidcpy(dest, kh_value(map, pos));
+ return 0;
+ }
+ return -1;
+}
+
int repo_loose_object_map_oid(struct repository *repo,
const struct object_id *src,
const struct git_hash_algo *to,
struct object_id *dest)
{
struct odb_source_files *files = odb_source_files_downcast(repo->objects->source);
- kh_oid_map_t *map;
- khiter_t pos;
for (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {
- struct loose_object_map *loose_map = dir->loose->map;
- if (!loose_map)
+ int ret;
+
+ if (!dir->loose)
continue;
- map = (to == repo->compat_hash_algo) ?
- loose_map->to_compat :
- loose_map->to_storage;
- pos = kh_get_oid_map(map, *src);
- if (pos < kh_end(map)) {
- oidcpy(dest, kh_value(map, pos));
- return 0;
- }
+ ret = source_loose_object_map_oid(dir->loose, src, to, dest);
+ if (ret != -1)
+ return ret;
}
return -1;
}
diff --git a/loose.h b/loose.h
index ed663ac550..f3b1e47363 100644
--- a/loose.h
+++ b/loose.h
@@ -24,4 +24,9 @@ int repo_add_loose_object_map(struct odb_source_loose *loose,
int repo_read_loose_object_map(struct repository *repo);
int repo_write_loose_object_map(struct repository *repo);
+int source_loose_object_map_oid(struct odb_source_loose *loose,
+ const struct object_id *src,
+ const struct git_hash_algo *to,
+ struct object_id *dest);
+
#endif
--
2.56.0.215.g47fdf2208c
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 02/13] loose: rename repo_add_loose_object_map
2026-10-23 7:25 [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Christian Couder
2026-10-23 7:25 ` [PATCH v2 01/13] loose: allow looking up in object map with source Christian Couder
@ 2026-10-23 7:25 ` Christian Couder
2026-10-23 7:25 ` [PATCH v2 03/13] loose: add a flag to control whether we write mapped OIDs Christian Couder
` (12 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Christian Couder @ 2026-10-23 7:25 UTC (permalink / raw)
To: git
Cc: Junio C Hamano, brian m . carlson, Patrick Steinhardt,
Elijah Newren, Jeff King, Johannes Schindelin, Christian Couder
From: "brian m. carlson" <sandals@crustytoothpaste.net>
The pointer it takes is not a struct repository, but a struct
odb_source_loose. Let's rename this and we'll use it in our object
database abstraction in a future commit.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Christian Couder <christian.couder@gmail.com>
---
loose.c | 6 +++---
loose.h | 6 +++---
odb/source-loose.c | 4 ++--
3 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/loose.c b/loose.c
index fe8f540979..8d7ea27448 100644
--- a/loose.c
+++ b/loose.c
@@ -210,9 +210,9 @@ static int write_one_object(struct odb_source_loose *loose,
return -1;
}
-int repo_add_loose_object_map(struct odb_source_loose *loose,
- const struct object_id *oid,
- const struct object_id *compat_oid)
+int source_add_loose_object_map(struct odb_source_loose *loose,
+ const struct object_id *oid,
+ const struct object_id *compat_oid)
{
int inserted = 0;
diff --git a/loose.h b/loose.h
index f3b1e47363..6f0c8554ca 100644
--- a/loose.h
+++ b/loose.h
@@ -18,9 +18,6 @@ int repo_loose_object_map_oid(struct repository *repo,
const struct object_id *src,
const struct git_hash_algo *dest_algo,
struct object_id *dest);
-int repo_add_loose_object_map(struct odb_source_loose *loose,
- const struct object_id *oid,
- const struct object_id *compat_oid);
int repo_read_loose_object_map(struct repository *repo);
int repo_write_loose_object_map(struct repository *repo);
@@ -28,5 +25,8 @@ int source_loose_object_map_oid(struct odb_source_loose *loose,
const struct object_id *src,
const struct git_hash_algo *to,
struct object_id *dest);
+int source_add_loose_object_map(struct odb_source_loose *loose,
+ const struct object_id *oid,
+ const struct object_id *compat_oid);
#endif
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 0664634938..3c0c85f3e3 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -848,7 +848,7 @@ static int odb_source_loose_write_object(struct odb_source *source,
return -1;
if (compat_oid)
- return repo_add_loose_object_map(loose, oid, compat_oid);
+ return source_add_loose_object_map(loose, oid, compat_oid);
return 0;
}
@@ -970,7 +970,7 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,
err = finalize_object_file_flags(loose->base.odb->repo, tmp_file.buf, filename.buf,
FOF_SKIP_COLLISION_CHECK);
if (!err && compat)
- err = repo_add_loose_object_map(loose, oid, &compat_oid);
+ err = source_add_loose_object_map(loose, oid, &compat_oid);
cleanup:
strbuf_release(&tmp_file);
strbuf_release(&filename);
--
2.56.0.215.g47fdf2208c
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 03/13] loose: add a flag to control whether we write mapped OIDs
2026-10-23 7:25 [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Christian Couder
2026-10-23 7:25 ` [PATCH v2 01/13] loose: allow looking up in object map with source Christian Couder
2026-10-23 7:25 ` [PATCH v2 02/13] loose: rename repo_add_loose_object_map Christian Couder
@ 2026-10-23 7:25 ` Christian Couder
2026-10-23 7:25 ` [PATCH v2 04/13] odb: implement interoperability functions Christian Couder
` (11 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Christian Couder @ 2026-10-23 7:25 UTC (permalink / raw)
To: git
Cc: Junio C Hamano, brian m . carlson, Patrick Steinhardt,
Elijah Newren, Jeff King, Johannes Schindelin, Christian Couder
From: "brian m. carlson" <sandals@crustytoothpaste.net>
Right now, we always write the object to the loose object map. However,
in the future, we'll need to temporarily store some object ID mappings
in memory so that we can rewrite other objects, such as when we're
indexing packs. Let's make this easier by setting a flag indicating
whether we want to write this to disk.
The bottom six bits will be used for an object type in a future commit,
so pick 1 << 6 as the write bit to avoid conflicts.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Christian Couder <christian.couder@gmail.com>
---
loose.c | 5 +++--
loose.h | 6 +++++-
odb/source-loose.c | 4 ++--
3 files changed, 10 insertions(+), 5 deletions(-)
diff --git a/loose.c b/loose.c
index 8d7ea27448..090681b4b3 100644
--- a/loose.c
+++ b/loose.c
@@ -212,7 +212,8 @@ static int write_one_object(struct odb_source_loose *loose,
int source_add_loose_object_map(struct odb_source_loose *loose,
const struct object_id *oid,
- const struct object_id *compat_oid)
+ const struct object_id *compat_oid,
+ int flags)
{
int inserted = 0;
@@ -220,7 +221,7 @@ int source_add_loose_object_map(struct odb_source_loose *loose,
return 0;
inserted = insert_loose_map(loose, oid, compat_oid);
- if (inserted)
+ if (inserted && (flags & OBJ_MAP_WRITE))
return write_one_object(loose, oid, compat_oid);
return 0;
}
diff --git a/loose.h b/loose.h
index 6f0c8554ca..19d3be26a3 100644
--- a/loose.h
+++ b/loose.h
@@ -3,6 +3,9 @@
#include "khash.h"
+/* Should we write this object to disk? */
+#define OBJ_MAP_WRITE (1 << 6)
+
struct repository;
struct odb_source_loose;
@@ -27,6 +30,7 @@ int source_loose_object_map_oid(struct odb_source_loose *loose,
struct object_id *dest);
int source_add_loose_object_map(struct odb_source_loose *loose,
const struct object_id *oid,
- const struct object_id *compat_oid);
+ const struct object_id *compat_oid,
+ int flags);
#endif
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 3c0c85f3e3..b58184f31d 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -848,7 +848,7 @@ static int odb_source_loose_write_object(struct odb_source *source,
return -1;
if (compat_oid)
- return source_add_loose_object_map(loose, oid, compat_oid);
+ return source_add_loose_object_map(loose, oid, compat_oid, OBJ_MAP_WRITE);
return 0;
}
@@ -970,7 +970,7 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,
err = finalize_object_file_flags(loose->base.odb->repo, tmp_file.buf, filename.buf,
FOF_SKIP_COLLISION_CHECK);
if (!err && compat)
- err = source_add_loose_object_map(loose, oid, &compat_oid);
+ err = source_add_loose_object_map(loose, oid, &compat_oid, OBJ_MAP_WRITE);
cleanup:
strbuf_release(&tmp_file);
strbuf_release(&filename);
--
2.56.0.215.g47fdf2208c
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 04/13] odb: implement interoperability functions
2026-10-23 7:25 [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Christian Couder
` (2 preceding siblings ...)
2026-10-23 7:25 ` [PATCH v2 03/13] loose: add a flag to control whether we write mapped OIDs Christian Couder
@ 2026-10-23 7:25 ` Christian Couder
2026-10-23 7:25 ` [PATCH v2 05/13] object-file-convert: implement mapping in terms of object DB Christian Couder
` (10 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Christian Couder @ 2026-10-23 7:25 UTC (permalink / raw)
To: git
Cc: Junio C Hamano, brian m . carlson, Patrick Steinhardt,
Elijah Newren, Jeff King, Johannes Schindelin, Christian Couder
From: "brian m. carlson" <sandals@crustytoothpaste.net>
An object database running with a compatibility algorithm needs a way
to map object IDs from one algorithm to another and to write mappings
into the repository. Add these functions and implement them for the
"files" backend and for the "loose" and "packed" sources it is made of,
so that we can use them in the future.
Like when reading objects, the "files" backend looks up mappings in all
of its object directories. It starts with the main one and continues
with alternates, which is what the loose object map code already does.
If a mapping is not found in a directory, the loose object map of that
directory is reloaded and checked again, as another process may have
written new mappings in the meantime. New mappings are only written
into the main object directory, which is also where new objects are
written.
Note that the "packed" sources don't write any information, since that
will be written when we index the pack, and they don't currently read
any information, since we don't yet have support for pack index v3,
which we'll need to read this properly. This functionality will come
in a future commit.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Christian Couder <christian.couder@gmail.com>
---
odb.c | 17 ++++++++++++
odb.h | 24 +++++++++++++++++
odb/source-files.c | 38 ++++++++++++++++++++++++++
odb/source-loose.c | 27 +++++++++++++++++++
odb/source-packed.c | 19 +++++++++++++
odb/source.h | 65 +++++++++++++++++++++++++++++++++++++++++++++
6 files changed, 190 insertions(+)
diff --git a/odb.c b/odb.c
index 98c5c862f5..a0a1be7c7e 100644
--- a/odb.c
+++ b/odb.c
@@ -590,6 +590,23 @@ int odb_count_objects(struct object_database *odb,
return ret;
}
+int odb_map_oid(struct object_database *odb,
+ const struct object_id *src,
+ const struct git_hash_algo *to,
+ struct object_id *dest,
+ enum odb_map_oid_flags flags)
+{
+ return odb_source_map_oid(odb->source, src, to, dest, flags);
+}
+
+int odb_write_oid_map(struct object_database *odb,
+ const struct object_id *oid,
+ const struct object_id *compat_oid,
+ enum odb_write_oid_map_flags flags)
+{
+ return odb_source_write_oid_map(odb->source, oid, compat_oid, flags);
+}
+
/*
* Return the slot of the most-significant bit set in "val". There are various
* ways to do this quickly with fls() or __builtin_clzl(), but speed is
diff --git a/odb.h b/odb.h
index 0ccc47ee73..70af2c7c78 100644
--- a/odb.h
+++ b/odb.h
@@ -2,6 +2,7 @@
#define ODB_H
#include "hashmap.h"
+#include "loose.h"
#include "object.h"
#include "oid-array.h"
#include "oidset.h"
@@ -649,6 +650,29 @@ static inline int odb_write_object(struct object_database *odb,
struct odb_stream;
+enum odb_map_oid_flags {
+ /* Reload this source to obtain any new information. */
+ ODB_MAP_OID_RELOAD = (1 << 0),
+};
+
+int odb_map_oid(struct object_database *odb,
+ const struct object_id *src,
+ const struct git_hash_algo *to,
+ struct object_id *dest,
+ enum odb_map_oid_flags flags);
+
+enum odb_write_oid_map_flags {
+ /*
+ * Serialize this data to disk instead of persisting it only in memory.
+ */
+ ODB_WRITE_OID_MAP_WRITE = OBJ_MAP_WRITE,
+};
+
+int odb_write_oid_map(struct object_database *odb,
+ const struct object_id *oid,
+ const struct object_id *compat_oid,
+ enum odb_write_oid_map_flags flags);
+
int odb_write_object_stream(struct object_database *odb,
struct odb_stream *stream,
struct object_id *oid);
diff --git a/odb/source-files.c b/odb/source-files.c
index 455f409db0..4f4f7f1f87 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -6,6 +6,7 @@
#include "gettext.h"
#include "hex.h"
#include "lockfile.h"
+#include "loose.h"
#include "object-file.h"
#include "odb.h"
#include "odb/source.h"
@@ -1234,6 +1235,41 @@ struct odb_files_dir *odb_source_files_find_dir(struct object_database *odb, con
return dir;
}
+static int odb_source_files_map_oid(struct odb_source *source,
+ const struct object_id *src,
+ const struct git_hash_algo *to,
+ struct object_id *dest,
+ enum odb_map_oid_flags flags)
+{
+ struct odb_source_files *files = odb_source_files_downcast(source);
+
+ for (struct odb_files_dir *dir = files->dirs; dir; dir = dir->next) {
+ int ret;
+
+ ret = odb_source_map_oid(&dir->loose->base, src, to, dest, flags);
+ if (ret != -1)
+ return ret;
+ ret = odb_source_map_oid(&dir->packed->base, src, to, dest, flags);
+ if (ret != -1)
+ return ret;
+ ret = odb_source_map_oid(&dir->loose->base, src, to, dest,
+ flags | ODB_MAP_OID_RELOAD);
+ if (ret != -1)
+ return ret;
+ }
+
+ return -1;
+}
+
+static inline int odb_source_files_write_oid_map(struct odb_source *source,
+ const struct object_id *oid,
+ const struct object_id *compat_oid,
+ enum odb_write_oid_map_flags flags)
+{
+ struct odb_source_files *files = odb_source_files_downcast(source);
+ return odb_source_write_oid_map(&files->dirs->loose->base, oid, compat_oid, flags);
+}
+
struct odb_source_files *odb_source_files_new(struct object_database *odb,
enum odb_new_flags flags)
{
@@ -1271,6 +1307,8 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
files->base.write_object = odb_source_files_write_object;
files->base.write_object_stream = odb_source_files_write_object_stream;
files->base.begin_transaction = odb_source_files_begin_transaction;
+ files->base.map_oid = odb_source_files_map_oid;
+ files->base.write_oid_map = odb_source_files_write_oid_map;
files->base.optimize = odb_source_files_optimize;
files->base.optimize_required = odb_source_files_optimize_required;
files->base.generate_pack = odb_source_files_generate_pack;
diff --git a/odb/source-loose.c b/odb/source-loose.c
index b58184f31d..22c46bd64a 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -1017,6 +1017,31 @@ static void odb_source_loose_reparent(const char *old_cwd,
loose->base.path = path;
}
+static int odb_source_loose_map_oid(struct odb_source *source, const struct object_id *src,
+ const struct git_hash_algo *to, struct object_id *dest,
+ enum odb_map_oid_flags flags)
+{
+ struct odb_source_loose *loose = odb_source_loose_downcast(source);
+ /*
+ * We may have loaded the object map at repo initialization but
+ * another process (perhaps upstream of a pipe from us) may have
+ * written a new object into the map. If the object is missing,
+ * let's reload the map to see if the object has appeared.
+ */
+ if (flags & ODB_MAP_OID_RELOAD)
+ loose_object_map_load(loose);
+ return source_loose_object_map_oid(loose, src, to, dest);
+}
+
+static inline int odb_source_loose_write_oid_map(struct odb_source *source,
+ const struct object_id *oid,
+ const struct object_id *compat_oid,
+ enum odb_write_oid_map_flags flags)
+{
+ struct odb_source_loose *loose = odb_source_loose_downcast(source);
+ return source_add_loose_object_map(loose, oid, compat_oid, flags);
+}
+
static void odb_source_loose_free(struct odb_source *source)
{
struct odb_source_loose *loose = odb_source_loose_downcast(source);
@@ -1139,6 +1164,8 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
loose->base.write_object = odb_source_loose_write_object;
loose->base.write_object_stream = odb_source_loose_write_object_stream;
loose->base.begin_transaction = odb_source_loose_begin_transaction;
+ loose->base.map_oid = odb_source_loose_map_oid;
+ loose->base.write_oid_map = odb_source_loose_write_oid_map;
if (!is_absolute_path(loose->base.path))
chdir_notify_register(odb_source_loose_reparent, loose);
diff --git a/odb/source-packed.c b/odb/source-packed.c
index 2226fbd313..ee1edfdb2b 100644
--- a/odb/source-packed.c
+++ b/odb/source-packed.c
@@ -863,6 +863,23 @@ static void odb_source_packed_close(struct odb_source *source)
packed->midx = NULL;
}
+static int odb_source_packed_map_oid(struct odb_source *source UNUSED,
+ const struct object_id *src UNUSED,
+ const struct git_hash_algo *to UNUSED,
+ struct object_id *dest UNUSED,
+ enum odb_map_oid_flags flags UNUSED)
+{
+ return -1;
+}
+
+static inline int odb_source_packed_write_oid_map(struct odb_source *source UNUSED,
+ const struct object_id *oid UNUSED,
+ const struct object_id *compat_oid UNUSED,
+ enum odb_write_oid_map_flags flags UNUSED)
+{
+ return -1;
+}
+
static void odb_source_packed_free(struct odb_source *source)
{
struct odb_source_packed *packed = odb_source_packed_downcast(source);
@@ -1013,6 +1030,8 @@ struct odb_source_packed *odb_source_packed_new(struct object_database *odb,
packed->base.write_object = odb_source_packed_write_object;
packed->base.write_object_stream = odb_source_packed_write_object_stream;
packed->base.begin_transaction = odb_source_packed_begin_transaction;
+ packed->base.map_oid = odb_source_packed_map_oid;
+ packed->base.write_oid_map = odb_source_packed_write_oid_map;
if (!is_absolute_path(path))
chdir_notify_register(odb_source_packed_reparent, packed);
diff --git a/odb/source.h b/odb/source.h
index 1867347dfc..e1f01359ce 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -300,6 +300,42 @@ struct odb_source {
*/
int (*fsck)(struct odb_source *source,
struct odb_fsck_options *options);
+
+ /*
+ * This callback is expected to provide the mapping of an object from
+ * the algorithm specified in src to the algorithm specified in to,
+ * leaving the result in dest. flags is a combination of `enum
+ * odb_map_oid_flags` values: with ODB_MAP_OID_RELOAD, the source should
+ * reload its mapping data before the lookup, as another process may
+ * have written new mappings in the meantime.
+ *
+ * The algo field in src must point to a valid algorithm (specifically,
+ * it must not be zero). to must likewise point to a valid algorithm;
+ * it must be non-NULL.
+ *
+ * The callback is expected to return 0 on success, -1 if the object is
+ * not found, and a different negative error code on error.
+ */
+ int (*map_oid)(struct odb_source *source, const struct object_id *src,
+ const struct git_hash_algo *to, struct object_id *dest,
+ enum odb_map_oid_flags flags);
+
+ /*
+ * This callback is expected to write a mapping of an object between oid
+ * and compat_oid. flags is a combination of `enum
+ * odb_write_oid_map_flags` values: with ODB_WRITE_OID_MAP_WRITE, the
+ * mapping should be persisted to disk; otherwise it may only be kept in
+ * memory.
+ *
+ * The algo fields in oid and compat_oid must point to valid algorithms
+ * (specifically, they must not be zero).
+ *
+ * The callback is expected to return 0 on success, and a negative
+ * error code on error.
+ */
+ int (*write_oid_map)(struct odb_source *source, const struct object_id *oid,
+ const struct object_id *compat_oid,
+ enum odb_write_oid_map_flags flags);
};
/*
@@ -555,4 +591,33 @@ static inline int odb_source_fsck(struct odb_source *source,
return source->fsck(source, opts);
}
+/*
+ * Map the specified object ID into the specified algorithm.
+ *
+ * Returns 0 on success, -1 if the object is not found, and a different negative
+ * error code otherwise.
+ */
+static inline int odb_source_map_oid(struct odb_source *source,
+ const struct object_id *src,
+ const struct git_hash_algo *to,
+ struct object_id *dest,
+ int flags)
+{
+ return source->map_oid(source, src, to, dest, flags);
+}
+
+/*
+ * Create a mapping between the two object IDs, which should be in the main and
+ * compatibility hash algorithms.
+ *
+ * Returns 0 on success, a negative error otherwise.
+ */
+static inline int odb_source_write_oid_map(struct odb_source *source,
+ const struct object_id *oid,
+ const struct object_id *compat_oid,
+ int flags)
+{
+ return source->write_oid_map(source, oid, compat_oid, flags);
+}
+
#endif
--
2.56.0.215.g47fdf2208c
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 05/13] object-file-convert: implement mapping in terms of object DB
2026-10-23 7:25 [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Christian Couder
` (3 preceding siblings ...)
2026-10-23 7:25 ` [PATCH v2 04/13] odb: implement interoperability functions Christian Couder
@ 2026-10-23 7:25 ` Christian Couder
2026-10-23 7:25 ` [PATCH v2 06/13] pack-write: add support for pack index v3 Christian Couder
` (9 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Christian Couder @ 2026-10-23 7:25 UTC (permalink / raw)
To: git
Cc: Junio C Hamano, brian m . carlson, Patrick Steinhardt,
Elijah Newren, Jeff King, Johannes Schindelin, Christian Couder
From: "brian m. carlson" <sandals@crustytoothpaste.net>
Now that we have odb_map_oid, let's implement repo_oid_to_algop in terms
of it. We still need the latter because it does some cleanup that needs
to happen so that our object database backend need not deal with these
issues.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Christian Couder <christian.couder@gmail.com>
---
object-file-convert.c | 14 ++------------
1 file changed, 2 insertions(+), 12 deletions(-)
diff --git a/object-file-convert.c b/object-file-convert.c
index 63ee18630b..90a75a6c21 100644
--- a/object-file-convert.c
+++ b/object-file-convert.c
@@ -7,6 +7,7 @@
#include "repository.h"
#include "hash.h"
#include "object.h"
+#include "odb.h"
#include "loose.h"
#include "commit.h"
#include "gpg-interface.h"
@@ -35,18 +36,7 @@ int repo_oid_to_algop(struct repository *repo, const struct object_id *srcoid,
oidcpy(dest, src);
return 0;
}
- if (repo_loose_object_map_oid(repo, src, to, dest)) {
- /*
- * We may have loaded the object map at repo initialization but
- * another process (perhaps upstream of a pipe from us) may have
- * written a new object into the map. If the object is missing,
- * let's reload the map to see if the object has appeared.
- */
- repo_read_loose_object_map(repo);
- if (repo_loose_object_map_oid(repo, src, to, dest))
- return -1;
- }
- return 0;
+ return odb_map_oid(repo->objects, src, to, dest, 0);
}
static int decode_tree_entry_raw(struct object_id *oid, const char **path,
--
2.56.0.215.g47fdf2208c
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 06/13] pack-write: add support for pack index v3
2026-10-23 7:25 [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Christian Couder
` (4 preceding siblings ...)
2026-10-23 7:25 ` [PATCH v2 05/13] object-file-convert: implement mapping in terms of object DB Christian Couder
@ 2026-10-23 7:25 ` Christian Couder
2026-10-23 7:25 ` [PATCH v2 07/13] builtin: populate pack index v3 fields when indexing Christian Couder
` (8 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Christian Couder @ 2026-10-23 7:25 UTC (permalink / raw)
To: git
Cc: Junio C Hamano, brian m . carlson, Patrick Steinhardt,
Elijah Newren, Jeff King, Johannes Schindelin, Christian Couder
From: "brian m. carlson" <sandals@crustytoothpaste.net>
As part of the SHA-256 transition plan, we'll be adding support for pack
index v3. This version supports multiple hash algorithms so we can
write both the SHA-256 and SHA-1 object names into the same repository.
To do so, let's add support for the compatibility object ID and the
entry number into the index entry structure. We then use this
information when writing the pack. There is no code to populate these
fields yet; this functionality will be coming in a future commit.
We always choose to write the minimum amount of padding that is
necessary to cause the end of the short object name table to end on a
multiple of 4. Git reads many tables as 32-bit values and if we didn't
do this, the operations would fail on architectures which don't support
unaligned access and would be slower on all other architectures.
We choose to always write the main hash algorithm first in the pack.
This optimizes for the case when we're interacting natively with the
remote side by permitting us to use the CRC32 entries.
Note that we never write index v3, since we cannot read it yet, and
doing so will break the tests. In a future commit, once we can read it,
we'll enable it by default when a compatibility algorithm is enabled.
Index v3 needs the objects both in pack order and sorted by object ID,
so when writing a v3 index, sort copies of the array instead of the
array itself. Callers, like the bitmap code in pack-objects, rely on
the array being sorted by object ID when write_idx_file() returns,
though, so sort it before returning in that case too.
When asked to reject duplicate objects using `--strict`, look for them
in the arrays sorted by object ID, where duplicates are adjacent.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Christian Couder <christian.couder@gmail.com>
---
pack-write.c | 263 +++++++++++++++++++++++++++++++++++++++++++++------
pack.h | 2 +
2 files changed, 235 insertions(+), 30 deletions(-)
diff --git a/pack-write.c b/pack-write.c
index 85674e4b72..1834394a77 100644
--- a/pack-write.c
+++ b/pack-write.c
@@ -22,13 +22,20 @@ void reset_pack_idx_option(struct pack_idx_option *opts)
opts->delta_base_cache_limit = DEFAULT_DELTA_BASE_CACHE_LIMIT;
}
-static int sha1_compare(const void *_a, const void *_b)
+static int oid_compare(const void *_a, const void *_b)
{
struct pack_idx_entry *a = *(struct pack_idx_entry **)_a;
struct pack_idx_entry *b = *(struct pack_idx_entry **)_b;
return oidcmp(&a->oid, &b->oid);
}
+static int oid_compare_compat(const void *_a, const void *_b)
+{
+ struct pack_idx_entry *a = *(struct pack_idx_entry **)_a;
+ struct pack_idx_entry *b = *(struct pack_idx_entry **)_b;
+ return oidcmp(&a->compat_oid, &b->compat_oid);
+}
+
static int cmp_uint32(const void *a_, const void *b_)
{
uint32_t a = *((uint32_t *)a_);
@@ -50,30 +57,156 @@ static int need_large_offset(off_t offset, const struct pack_idx_option *opts)
sizeof(ofsval), cmp_uint32);
}
+static inline int last_matching_offset(const struct object_id *a, const struct object_id *b,
+ const struct git_hash_algo *algop)
+{
+ size_t i;
+ for (i = 0; i < algop->rawsz; i++)
+ if (a->hash[i] != b->hash[i])
+ return i;
+ /* We should never hit this case. */
+ return i;
+}
+
+static const struct object_id *main_oid(const struct pack_idx_entry *ent)
+{
+ return &ent->oid;
+}
+
+static const struct object_id *compat_oid(const struct pack_idx_entry *ent)
+{
+ return &ent->compat_oid;
+}
+
+static uint32_t find_short_name_len(struct pack_idx_entry **sorted, size_t nr_objects,
+ const struct git_hash_algo *algop,
+ const struct object_id *(*fn)(const struct pack_idx_entry *))
+{
+ size_t i, len = 1;
+ for (i = 1; i < nr_objects; i++) {
+ const struct object_id *oid1 = fn(sorted[i - 1]);
+ const struct object_id *oid2 = fn(sorted[i]);
+ size_t offset = last_matching_offset(oid1, oid2, algop);
+
+ if (offset >= len)
+ len = offset + 1;
+ }
+ return len > algop->rawsz ? algop->rawsz : len;
+}
+
+struct pack_idx_format_header {
+ uint32_t id;
+ uint32_t short_name_len;
+ uint64_t offset;
+};
+
+/* This struct contains fields beyond the signature and version number. */
+struct pack_idx_header_v3 {
+ uint32_t hdr_len;
+ uint32_t nr_objects;
+ uint32_t nr_formats;
+};
+
+static void write_v3_header(struct repository *repo,
+ int nr_objects, int nr_large_offset,
+ uint32_t short_name_len[2], int padding[2],
+ struct hashfile *f)
+{
+ struct pack_idx_header_v3 hdr;
+ struct pack_idx_format_header format[2];
+ const struct git_hash_algo *compat = repo->compat_hash_algo;
+ uint32_t nformats = compat ? 2 : 1;
+ /*
+ * signature (4) + version (4) + length (4) + nobjects(4) + nformats (4) +
+ * each format: identifier (4) + length of shortened names (4) +
+ * table offset (8) +
+ * offset to trailer (8)
+ */
+ uint32_t hdrlen = 4 + 4 + 4 + 4 + 4 + nformats * (4 + 4 + 8) + 8;
+ uint64_t offset = hdrlen;
+ uint64_t short_table = short_name_len[0] * nr_objects;
+
+ hdr.hdr_len = htonl(hdrlen);
+ hdr.nr_objects = htonl(nr_objects);
+ hdr.nr_formats = htonl(nformats);
+ hashwrite(f, &hdr, sizeof(hdr));
+
+ /* padding */
+ if (short_table & 3) {
+ padding[0] = 4 - (short_table & 3);
+ offset += padding[0];
+ }
+
+ format[0].id = htonl(repo->hash_algo->format_id);
+ format[0].offset = htonll(offset);
+ format[0].short_name_len = htonl(short_name_len[0]);
+ hashwrite(f, &format[0], sizeof(format[0]));
+ offset += (uint64_t)(short_name_len[0] + repo->hash_algo->rawsz + 4 + 4 + 4) * nr_objects +
+ 8 * nr_large_offset;
+
+ if (compat) {
+ short_table = short_name_len[1] * nr_objects;
+ /* padding */
+ if (short_table & 3) {
+ padding[1] = 4 - (short_table & 3);
+ offset += padding[1];
+ }
+
+ format[1].id = htonl(compat->format_id);
+ format[1].offset = htonll(offset);
+ format[1].short_name_len = htonl(short_name_len[1]);
+ hashwrite(f, &format[1], sizeof(format[1]));
+ offset += (uint64_t)(short_name_len[1] + compat->rawsz + 4) * nr_objects;
+ }
+
+ offset = htonll(offset);
+ hashwrite(f, &offset, sizeof(offset));
+}
+
/*
- * The *sha1 contains the pack content SHA1 hash.
- * The objects array passed in will be sorted by SHA1 on exit.
+ * The *hash contains the pack content hash.
+ * The objects array passed in should be in pack order and will be sorted by
+ * object ID on return.
*/
const char *write_idx_file(struct repository *repo,
const char *index_name, struct pack_idx_entry **objects,
int nr_objects, const struct pack_idx_option *opts,
- const unsigned char *sha1)
+ const unsigned char *hash)
{
struct hashfile *f;
- struct pack_idx_entry **sorted_by_sha, **list, **last;
- off_t last_obj_offset = 0;
+ struct pack_idx_entry **sorted_by_sha, **sorted_by_sha_compat = NULL, **list, **last;
int i, fd;
uint32_t index_version;
+ unsigned int nr_large_offset = 0;
+ const struct git_hash_algo *compat = repo->compat_hash_algo;
+ uint32_t short_name_len[2] = { 0 };
+ int padding_size[2] = { 0 };
+ uint8_t zero[4] = { 0 };
+
+ index_version = opts->version;
if (nr_objects) {
sorted_by_sha = objects;
list = sorted_by_sha;
last = sorted_by_sha + nr_objects;
for (i = 0; i < nr_objects; ++i) {
- if (objects[i]->offset > last_obj_offset)
- last_obj_offset = objects[i]->offset;
+ objects[i]->idx = i;
+ if (need_large_offset(objects[i]->offset, opts))
+ nr_large_offset++;
}
- QSORT(sorted_by_sha, nr_objects, sha1_compare);
+ /*
+ * If we're using index v3, we need both pack order and sorted
+ * order.
+ */
+ if (index_version >= 3) {
+ DUP_ARRAY(sorted_by_sha, objects, nr_objects);
+ if (compat) {
+ DUP_ARRAY(sorted_by_sha_compat, objects,
+ nr_objects);
+ QSORT(sorted_by_sha_compat, nr_objects, oid_compare_compat);
+ }
+ }
+ QSORT(sorted_by_sha, nr_objects, oid_compare);
}
else
sorted_by_sha = list = last = NULL;
@@ -95,7 +228,8 @@ const char *write_idx_file(struct repository *repo,
}
/* if last object's offset is >= 2^31 we should use index V2 */
- index_version = need_large_offset(last_obj_offset, opts) ? 2 : opts->version;
+ if (index_version == 1 && nr_large_offset)
+ index_version = 2;
/* index versions 2 and above need a header */
if (index_version >= 2) {
@@ -105,50 +239,84 @@ const char *write_idx_file(struct repository *repo,
hashwrite(f, &hdr, sizeof(hdr));
}
- /*
- * Write the first-level table (the list is sorted,
- * but we use a 256-entry lookup to be able to avoid
- * having to do eight extra binary search iterations).
- */
- for (i = 0; i < 256; i++) {
- struct pack_idx_entry **next = list;
- while (next < last) {
- struct pack_idx_entry *obj = *next;
- if (obj->oid.hash[0] != i)
- break;
- next++;
+ if (index_version >= 3) {
+ short_name_len[0] = find_short_name_len(sorted_by_sha,
+ nr_objects,
+ repo->hash_algo,
+ main_oid);
+ if (compat)
+ short_name_len[1] = find_short_name_len(sorted_by_sha_compat,
+ nr_objects,
+ compat, compat_oid);
+ write_v3_header(repo, nr_objects, nr_large_offset, short_name_len, padding_size, f);
+ hashwrite(f, zero, padding_size[0]);
+
+ /* Write the sorted abbreviated names */
+ list = sorted_by_sha;
+ for (i = 0; i < nr_objects; i++) {
+ struct pack_idx_entry *obj = *list++;
+ hashwrite(f, obj->oid.hash, short_name_len[0]);
+ if ((opts->flags & WRITE_IDX_STRICT) &&
+ (i && oideq(&list[-2]->oid, &obj->oid)))
+ die("The same object %s appears twice in the pack",
+ oid_to_hex(&obj->oid));
+ }
+ } else {
+ /*
+ * Write the first-level table (the list is sorted,
+ * but we use a 256-entry lookup to be able to avoid
+ * having to do eight extra binary search iterations).
+ */
+ for (i = 0; i < 256; i++) {
+ struct pack_idx_entry **next = list;
+ while (next < last) {
+ struct pack_idx_entry *obj = *next;
+ if (obj->oid.hash[0] != i)
+ break;
+ next++;
+ }
+ hashwrite_be32(f, next - sorted_by_sha);
+ list = next;
}
- hashwrite_be32(f, next - sorted_by_sha);
- list = next;
}
/*
* Write the actual SHA1 entries..
*/
- list = sorted_by_sha;
+ list = objects;
for (i = 0; i < nr_objects; i++) {
struct pack_idx_entry *obj = *list++;
if (index_version < 2)
hashwrite_be32(f, obj->offset);
hashwrite(f, obj->oid.hash, repo->hash_algo->rawsz);
- if ((opts->flags & WRITE_IDX_STRICT) &&
+ if ((opts->flags & WRITE_IDX_STRICT) && index_version < 3 &&
(i && oideq(&list[-2]->oid, &obj->oid)))
die("The same object %s appears twice in the pack",
oid_to_hex(&obj->oid));
}
+ if (index_version >= 3) {
+ /* write the sorted-to-pack index */
+ list = sorted_by_sha;
+ for (i = 0; i < nr_objects; i++) {
+ struct pack_idx_entry *obj = *list++;
+ uint32_t index = htonl(obj->idx);
+ hashwrite(f, &index, 4);
+ }
+ }
+
if (index_version >= 2) {
unsigned int nr_large_offset = 0;
/* write the crc32 table */
- list = sorted_by_sha;
+ list = objects;
for (i = 0; i < nr_objects; i++) {
struct pack_idx_entry *obj = *list++;
hashwrite_be32(f, obj->crc32);
}
/* write the 32-bit offset table */
- list = sorted_by_sha;
+ list = objects;
for (i = 0; i < nr_objects; i++) {
struct pack_idx_entry *obj = *list++;
uint32_t offset;
@@ -160,7 +328,7 @@ const char *write_idx_file(struct repository *repo,
}
/* write the large offset table */
- list = sorted_by_sha;
+ list = objects;
while (nr_large_offset) {
struct pack_idx_entry *obj = *list++;
uint64_t offset = obj->offset;
@@ -172,10 +340,45 @@ const char *write_idx_file(struct repository *repo,
}
}
- hashwrite(f, sha1, repo->hash_algo->rawsz);
+ if (index_version >= 3 && compat) {
+ hashwrite(f, zero, padding_size[1]);
+
+ /* Write the sorted abbreviated names */
+ list = sorted_by_sha_compat;
+ for (i = 0; i < nr_objects; i++) {
+ struct pack_idx_entry *obj = *list++;
+ hashwrite(f, obj->compat_oid.hash, short_name_len[1]);
+ if ((opts->flags & WRITE_IDX_STRICT) &&
+ (i && oideq(&list[-2]->compat_oid, &obj->compat_oid)))
+ die("The same object %s appears twice in the pack",
+ oid_to_hex(&obj->compat_oid));
+ }
+
+ list = objects;
+ for (i = 0; i < nr_objects; i++) {
+ struct pack_idx_entry *obj = *list++;
+ hashwrite(f, obj->compat_oid.hash, compat->rawsz);
+ }
+
+ /* write the sorted-to-pack index */
+ list = sorted_by_sha_compat;
+ for (i = 0; i < nr_objects; i++) {
+ struct pack_idx_entry *obj = *list++;
+ uint32_t index = htonl(obj->idx);
+ hashwrite(f, &index, 4);
+ }
+ }
+
+ hashwrite(f, hash, repo->hash_algo->rawsz);
finalize_hashfile(f, NULL, FSYNC_COMPONENT_PACK_METADATA,
CSUM_HASH_IN_STREAM | CSUM_CLOSE |
((opts->flags & WRITE_IDX_VERIFY) ? 0 : CSUM_FSYNC));
+
+ if (sorted_by_sha != objects) {
+ QSORT(objects, nr_objects, oid_compare);
+ free(sorted_by_sha);
+ }
+ free(sorted_by_sha_compat);
return index_name;
}
diff --git a/pack.h b/pack.h
index ada506b5c5..86b2a8ba7b 100644
--- a/pack.h
+++ b/pack.h
@@ -79,7 +79,9 @@ struct pack_idx_header {
*/
struct pack_idx_entry {
struct object_id oid;
+ struct object_id compat_oid;
uint32_t crc32;
+ uint32_t idx;
off_t offset;
};
--
2.56.0.215.g47fdf2208c
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 07/13] builtin: populate pack index v3 fields when indexing
2026-10-23 7:25 [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Christian Couder
` (5 preceding siblings ...)
2026-10-23 7:25 ` [PATCH v2 06/13] pack-write: add support for pack index v3 Christian Couder
@ 2026-10-23 7:25 ` Christian Couder
2026-10-23 7:25 ` [PATCH v2 08/13] object-file: handle compatibility algorithm Christian Couder
` (7 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Christian Couder @ 2026-10-23 7:25 UTC (permalink / raw)
To: git
Cc: Junio C Hamano, brian m . carlson, Patrick Steinhardt,
Elijah Newren, Jeff King, Johannes Schindelin, Christian Couder
From: "brian m. carlson" <sandals@crustytoothpaste.net>
Right now, we have support for writing pack index v3. However, we need
to populate the compatibility object ID in order to write it into the
repository. Let's do so by converting the objects from SHA-256 to SHA-1
as part of our indexing process.
Take advantage of the fact that a SHA-256 blob and its corresponding
SHA-1 blob are identical by always hashing them at the same time. For
non-blobs, we convert the objects from SHA-256 to SHA-1 before hashing.
Since, when converting an object, we may need to know about the
compatibility object IDs of objects earlier in the pack, on which they
depend, store the compatibility object IDs we generate in the loose
object map, but don't write them to disk. This is the simplest way to
store these objects and have them properly looked up by the same process
without polluting the loose object index on disk with all of these
objects.
Because these operations may be threaded and modifying the loose object
map isn't thread safe, add a mutex and use it to protect this structure.
Note that the object conversion routines use this structure. Acquire
and release the mutex between the conversion and writing to the map
because hashing may be slow.
Add a test in t1016 to check that index-pack can write a v3 index in a
repository with a compatibility algorithm, and a test that it rejects
packs with duplicate objects when writing a v3 index with `--strict`.
Also add a create_compat_repo() helper in t1016 to refactor common
code.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Christian Couder <christian.couder@gmail.com>
---
builtin/index-pack.c | 105 +++++++++++++++++++++++++++---
builtin/pack-objects.c | 8 ++-
t/t1016-compatObjectFormat.sh | 29 ++++++++-
t/t5308-pack-detect-duplicates.sh | 7 ++
4 files changed, 137 insertions(+), 12 deletions(-)
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 0264c18c5f..723595ae03 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -31,6 +31,8 @@
#include "run-command.h"
#include "setup.h"
#include "strvec.h"
+#include "loose.h"
+#include "object-file-convert.h"
static const char index_pack_usage[] =
"git index-pack [-v] [-o <index-file>] [--keep | --keep=<msg>] [--[no-]rev-index] [--verify] [--strict[=<msg-id>=<severity>...]] [--fsck-objects[=<msg-id>=<severity>...]] (<pack-file> | --stdin [--fix-thin] [<pack-file>])";
@@ -182,6 +184,10 @@ static pthread_mutex_t deepest_delta_mutex;
#define deepest_delta_lock() lock_mutex(&deepest_delta_mutex)
#define deepest_delta_unlock() unlock_mutex(&deepest_delta_mutex)
+static pthread_mutex_t repo_mutex;
+#define repo_lock() lock_mutex(&repo_mutex)
+#define repo_unlock() unlock_mutex(&repo_mutex)
+
static pthread_key_t key;
static inline void lock_mutex(pthread_mutex_t *mutex)
@@ -205,6 +211,7 @@ static void init_thread(void)
init_recursive_mutex(&read_mutex);
pthread_mutex_init(&counter_mutex, NULL);
pthread_mutex_init(&work_mutex, NULL);
+ pthread_mutex_init(&repo_mutex, NULL);
if (show_stat)
pthread_mutex_init(&deepest_delta_mutex, NULL);
pthread_key_create(&key, NULL);
@@ -225,6 +232,7 @@ static void cleanup_thread(void)
pthread_mutex_destroy(&read_mutex);
pthread_mutex_destroy(&counter_mutex);
pthread_mutex_destroy(&work_mutex);
+ pthread_mutex_destroy(&repo_mutex);
if (show_stat)
pthread_mutex_destroy(&deepest_delta_mutex);
for (i = 0; i < nr_threads; i++)
@@ -469,13 +477,15 @@ static int is_delta_type(enum object_type type)
}
static void *unpack_entry_data(off_t offset, size_t size,
- enum object_type type, struct object_id *oid)
+ enum object_type type, struct object_id *oid,
+ struct object_id *compat_oid)
{
static char fixed_buf[8192];
int status;
git_zstream stream;
void *buf;
- struct git_hash_ctx c;
+ struct git_hash_ctx c, c_compat;
+ const struct git_hash_algo *compat = the_repository->compat_hash_algo;
char hdr[32];
int hdrlen;
@@ -483,8 +493,15 @@ static void *unpack_entry_data(off_t offset, size_t size,
hdrlen = format_object_header(hdr, sizeof(hdr), type, size);
git_hash_init(&c, the_hash_algo);
git_hash_update(&c, hdr, hdrlen);
- } else
+ if (compat && compat_oid) {
+ git_hash_init(&c_compat, compat);
+ if (type == OBJ_BLOB)
+ git_hash_update(&c_compat, hdr, hdrlen);
+ }
+ } else {
oid = NULL;
+ compat_oid = NULL;
+ }
if (type == OBJ_BLOB &&
size > repo_settings_get_big_file_threshold(the_repository))
buf = fixed_buf;
@@ -504,6 +521,8 @@ static void *unpack_entry_data(off_t offset, size_t size,
use(input_len - stream.avail_in);
if (oid)
git_hash_update(&c, last_out, stream.next_out - last_out);
+ if (compat && compat_oid && type == OBJ_BLOB)
+ git_hash_update(&c_compat, last_out, stream.next_out - last_out);
if (buf == fixed_buf) {
stream.next_out = buf;
stream.avail_out = sizeof(fixed_buf);
@@ -514,13 +533,43 @@ static void *unpack_entry_data(off_t offset, size_t size,
git_inflate_end(&stream);
if (oid)
git_hash_final_oid(oid, &c);
+ if (compat && compat_oid) {
+ if (type != OBJ_BLOB) {
+ /*
+ * This is not a blob. Let's remap it to the proper
+ * algorithm.
+ */
+ struct strbuf outbuf = STRBUF_INIT;
+ int ret;
+ repo_lock();
+ ret = convert_object_file(the_repository, &outbuf,
+ the_repository->hash_algo,
+ compat,
+ buf, size, type, 0);
+ repo_unlock();
+ if (ret < 0)
+ bad_object(offset, _("could not convert object"));
+
+ hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %"PRIuMAX,
+ type_name(type),(uintmax_t)outbuf.len) + 1;
+
+ git_hash_update(&c_compat, hdr, hdrlen);
+ git_hash_update(&c_compat, outbuf.buf, outbuf.len);
+ strbuf_release(&outbuf);
+ }
+ git_hash_final_oid(compat_oid, &c_compat);
+ repo_lock();
+ odb_write_oid_map(the_repository->objects, oid, compat_oid, 0);
+ repo_unlock();
+ }
return buf == fixed_buf ? NULL : buf;
}
static void *unpack_raw_entry(struct object_entry *obj,
off_t *ofs_offset,
struct object_id *ref_oid,
- struct object_id *oid)
+ struct object_id *oid,
+ struct object_id *compat_oid)
{
unsigned char *p;
size_t size, c;
@@ -582,7 +631,7 @@ static void *unpack_raw_entry(struct object_entry *obj,
}
obj->hdr_size = consumed_bytes - obj->idx.offset;
- data = unpack_entry_data(obj->idx.offset, obj->size, obj->type, oid);
+ data = unpack_entry_data(obj->idx.offset, obj->size, obj->type, oid, compat_oid);
obj->idx.crc32 = input_crc32;
return data;
}
@@ -1069,6 +1118,45 @@ static struct base_data *resolve_delta(struct object_entry *delta_obj,
bad_object(delta_obj->idx.offset, _("failed to apply delta"));
hash_object_file(the_hash_algo, result_data, result_size,
delta_obj->real_type, &delta_obj->idx.oid);
+ if (the_repository->compat_hash_algo) {
+ struct strbuf outbuf = STRBUF_INIT;
+ int ret = 0;
+ char *buf;
+ size_t len;
+
+ if (delta_obj->real_type != OBJ_BLOB) {
+ repo_lock();
+ ret = convert_object_file(the_repository, &outbuf,
+ the_repository->hash_algo,
+ the_repository->compat_hash_algo,
+ result_data, result_size,
+ delta_obj->real_type, 0);
+ repo_unlock();
+ buf = outbuf.buf;
+ len = outbuf.len;
+ } else {
+ buf = result_data;
+ len = result_size;
+ }
+ if (ret < 0)
+ bad_object(delta_obj->idx.offset,
+ _("could not convert object from delta"));
+
+ hash_object_file(the_repository->compat_hash_algo, buf, len,
+ delta_obj->real_type, &delta_obj->idx.compat_oid);
+ /*
+ * We add this entry to the loose object map but do not write it
+ * to disk because we may need it available when mapping other
+ * objects. Storing it temporarily in the loose object map lets
+ * us keep it around for that purpose.
+ */
+ repo_lock();
+ odb_write_oid_map(the_repository->objects,
+ &delta_obj->idx.oid,
+ &delta_obj->idx.compat_oid, 0);
+ repo_unlock();
+ strbuf_release(&outbuf);
+ }
sha1_object(result_data, NULL, result_size, delta_obj->real_type,
&delta_obj->idx.oid);
@@ -1265,7 +1353,8 @@ static void parse_pack_objects(unsigned char *hash)
struct object_entry *obj = &objects[i];
void *data = unpack_raw_entry(obj, &ofs_delta->offset,
&ref_delta_oid,
- &obj->idx.oid);
+ &obj->idx.oid,
+ &obj->idx.compat_oid);
obj->real_type = obj->type;
if (obj->type == OBJ_OFS_DELTA) {
nr_ofs_deltas++;
@@ -1669,7 +1758,7 @@ static int git_index_pack_config(const char *k, const char *v,
if (!strcmp(k, "pack.indexversion")) {
opts->version = git_config_int(k, v, ctx->kvi);
- if (opts->version > 2)
+ if (opts->version > 3)
die(_("bad pack.indexVersion=%"PRIu32), opts->version);
return 0;
}
@@ -1991,7 +2080,7 @@ int cmd_index_pack(int argc,
} else if (starts_with(arg, "--index-version=")) {
char *c;
opts.version = strtoul(arg + 16, &c, 10);
- if (opts.version > 2)
+ if (opts.version > 3)
die(_("bad %s"), arg);
if (*c == ',')
opts.off32_limit = strtoul(c+1, &c, 0);
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index ca3a891dfb..fc4adb6877 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -47,6 +47,9 @@
#include "blob.h"
#include "tree.h"
#include "path-walk.h"
+#include "trace2.h"
+#include "loose.h"
+#include "object-file-convert.h"
/*
* Objects we are going to pack are collected in the `to_pack` structure.
@@ -1847,6 +1850,9 @@ static struct object_entry *create_object_entry(const struct object_id *oid,
entry = packlist_alloc(&to_pack, oid);
entry->hash = hash;
+ if (repo_oid_to_algop(the_repository, &entry->idx.oid,
+ the_repository->compat_hash_algo, &entry->idx.compat_oid) < 0)
+ die(_("can't map object %s while writing pack"), oid_to_hex(oid));
oe_set_type(entry, type);
if (exclude)
entry->preferred_base = 1;
@@ -5039,7 +5045,7 @@ static int option_parse_index_version(const struct option *opt,
BUG_ON_OPT_NEG(unset);
popts->version = strtoul(val, &c, 10);
- if (popts->version > 2)
+ if (popts->version > 3)
die(_("unsupported index version %s"), val);
if (*c == ',' && c[1])
popts->off32_limit = strtoul(c+1, &c, 0);
diff --git a/t/t1016-compatObjectFormat.sh b/t/t1016-compatObjectFormat.sh
index 9cafcee509..1560b6a2e2 100755
--- a/t/t1016-compatObjectFormat.sh
+++ b/t/t1016-compatObjectFormat.sh
@@ -45,6 +45,15 @@ compat_hash () {
esac
}
+create_compat_repo () {
+ local hash="$1"
+ local repo="$2"
+
+ git init --object-format="$hash" "$repo" &&
+ git -C "$repo" config set core.repositoryformatversion 1 &&
+ git -C "$repo" config set extensions.compatObjectFormat $(compat_hash "$hash")
+}
+
hello_oid () {
case "$1" in
"sha1")
@@ -192,9 +201,7 @@ do
for repo in alt borrow
do
test_when_finished "rm -rf $repo" &&
- git init --object-format=$hash $repo &&
- git -C $repo config set core.repositoryformatversion 1 &&
- git -C $repo config set extensions.compatObjectFormat $(compat_hash $hash) || exit 1
+ create_compat_repo $hash $repo || exit 1
done &&
git -C alt commit --allow-empty --message A &&
@@ -205,6 +212,22 @@ do
git -C borrow rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
test_cmp expect actual
'
+
+ test_expect_success 'index-pack writes pack index v3' '
+ test_when_finished "rm -rf v3-repo v3.idx" &&
+ create_compat_repo $hash v3-repo &&
+ echo content >v3-repo/file &&
+ git -C v3-repo add file &&
+ git -C v3-repo commit --message A &&
+ git -C v3-repo commit --allow-empty --message B &&
+ git -C v3-repo repack -a -d &&
+ pack=$(echo v3-repo/.git/objects/pack/pack-*.pack) &&
+ git -C v3-repo index-pack --index-version=3 \
+ -o "$(pwd)/v3.idx" "$(pwd)/$pack" &&
+ echo "ff 74 4f 63 00 00 00 03 " >expect &&
+ test_copy_bytes 8 <v3.idx | test-tool hexdump >actual &&
+ test_cmp expect actual
+ '
done
cd "$base"
diff --git a/t/t5308-pack-detect-duplicates.sh b/t/t5308-pack-detect-duplicates.sh
index 0f84137867..7b43db870a 100755
--- a/t/t5308-pack-detect-duplicates.sh
+++ b/t/t5308-pack-detect-duplicates.sh
@@ -79,4 +79,11 @@ test_expect_success 'index-pack can reject packs with duplicates' '
test_expect_code 1 git cat-file -e $LO_SHA1
'
+test_expect_success 'index-pack can reject packs with duplicates with index v3' '
+ clear_packs &&
+ create_pack dups.pack 2 &&
+ test_must_fail git index-pack --strict --index-version=3 --stdin <dups.pack &&
+ test_expect_code 1 git cat-file -e $LO_SHA1
+'
+
test_done
--
2.56.0.215.g47fdf2208c
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 08/13] object-file: handle compatibility algorithm
2026-10-23 7:25 [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Christian Couder
` (6 preceding siblings ...)
2026-10-23 7:25 ` [PATCH v2 07/13] builtin: populate pack index v3 fields when indexing Christian Couder
@ 2026-10-23 7:25 ` Christian Couder
2026-10-23 7:25 ` [PATCH v2 09/13] packfile: add support for reading pack index v3 Christian Couder
` (6 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Christian Couder @ 2026-10-23 7:25 UTC (permalink / raw)
To: git
Cc: Junio C Hamano, brian m . carlson, Patrick Steinhardt,
Elijah Newren, Jeff King, Johannes Schindelin, Christian Couder
From: "brian m. carlson" <sandals@crustytoothpaste.net>
When we check in a large blob, we stream it directly to a pack and then
write an index. However, in compatibility mode, we also need to make
sure to compute the object ID for the compatibility algorithm as well as
the main algorithm or we will incorrectly store the null OID in the
index we create. Do so here to avoid corrupting the repository.
Add a test that adds a large file in a repository with a compatibility
algorithm.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Christian Couder <christian.couder@gmail.com>
---
object-file.c | 18 ++++++++++++++++--
t/t1016-compatObjectFormat.sh | 14 ++++++++++++++
2 files changed, 30 insertions(+), 2 deletions(-)
diff --git a/object-file.c b/object-file.c
index c049bf13f7..b4c4a2ea0f 100644
--- a/object-file.c
+++ b/object-file.c
@@ -747,6 +747,7 @@ static int hash_stream(struct odb_stream *stream,
*/
static void stream_to_pack(struct transaction_packfile *state,
struct git_hash_ctx *ctx,
+ struct git_hash_ctx *compat_ctx,
struct odb_stream *stream)
{
git_zstream s;
@@ -774,6 +775,9 @@ static void stream_to_pack(struct transaction_packfile *state,
is_finished = true;
git_hash_update(ctx, ibuf, rsize);
+ if (compat_ctx)
+ git_hash_update(compat_ctx,
+ ibuf, rsize);
s.next_in = ibuf;
s.avail_in = rsize;
@@ -882,16 +886,22 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas
struct odb_transaction_files,
base);
struct transaction_packfile *state = &transaction->packfile;
- struct git_hash_ctx ctx;
+ struct git_hash_ctx ctx, compat_ctx;
+ struct object_id compat_oid;
unsigned char obuf[16384];
unsigned header_len;
struct hashfile_checkpoint checkpoint;
struct pack_idx_entry *idx;
+ const struct git_hash_algo *compat = transaction->base.source->odb->repo->compat_hash_algo;
header_len = format_object_header((char *)obuf, sizeof(obuf),
stream->type, stream->size);
git_hash_init(&ctx, transaction->base.source->odb->repo->hash_algo);
git_hash_update(&ctx, obuf, header_len);
+ if (compat) {
+ git_hash_init(&compat_ctx, compat);
+ git_hash_update(&compat_ctx, obuf, header_len);
+ }
/*
* If writing another object to the packfile could result in it
@@ -916,8 +926,10 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas
hashfile_checkpoint(state->f, &checkpoint);
idx->offset = state->offset;
crc32_begin(state->f);
- stream_to_pack(state, &ctx, stream);
+ stream_to_pack(state, &ctx, compat ? &compat_ctx : NULL, stream);
git_hash_final_oid(result_oid, &ctx);
+ if (compat)
+ git_hash_final_oid(&compat_oid, &compat_ctx);
idx->crc32 = crc32_end(state->f);
if (already_written(transaction, result_oid)) {
@@ -926,6 +938,8 @@ static int odb_transaction_files_write_object_stream(struct odb_transaction *bas
free(idx);
} else {
oidcpy(&idx->oid, result_oid);
+ if (compat)
+ oidcpy(&idx->compat_oid, &compat_oid);
ALLOC_GROW(state->written,
state->nr_written + 1,
state->alloc_written);
diff --git a/t/t1016-compatObjectFormat.sh b/t/t1016-compatObjectFormat.sh
index 1560b6a2e2..af9532aa1f 100755
--- a/t/t1016-compatObjectFormat.sh
+++ b/t/t1016-compatObjectFormat.sh
@@ -228,6 +228,20 @@ do
test_copy_bytes 8 <v3.idx | test-tool hexdump >actual &&
test_cmp expect actual
'
+
+ test_expect_success 'add a large file streamed into a pack' '
+ test_when_finished "rm -rf large-repo" &&
+ create_compat_repo $hash large-repo &&
+ git -C large-repo config set core.bigFileThreshold 1k &&
+ test-tool genrandom large 5000 >large-repo/large &&
+ git -C large-repo add large &&
+ ls large-repo/.git/objects/pack/pack-*.pack &&
+ git -C large-repo hash-object large >expect &&
+ git -C large-repo rev-parse :large >actual &&
+ test_cmp expect actual &&
+ git -C large-repo cat-file blob :large >actual &&
+ test_cmp large-repo/large actual
+ '
done
cd "$base"
--
2.56.0.215.g47fdf2208c
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 09/13] packfile: add support for reading pack index v3
2026-10-23 7:25 [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Christian Couder
` (7 preceding siblings ...)
2026-10-23 7:25 ` [PATCH v2 08/13] object-file: handle compatibility algorithm Christian Couder
@ 2026-10-23 7:25 ` Christian Couder
2026-10-23 7:25 ` [PATCH v2 10/13] packfile: validate pack index v3 tables Christian Couder
` (5 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Christian Couder @ 2026-10-23 7:25 UTC (permalink / raw)
To: git
Cc: Junio C Hamano, brian m . carlson, Patrick Steinhardt,
Elijah Newren, Jeff King, Johannes Schindelin, Christian Couder
From: "brian m. carlson" <sandals@crustytoothpaste.net>
Now that we can write pack index v3, we'll probably also want to read
it. Let's add support for that by making our bsearch function more
generic to handle lookups of any size and then using that both for older
version and for pack index v3.
To make lookups easier, let's store several offsets we compute from
values per format so we don't need to look them up when reading. Since
our pack may be larger than 4 GiB, let's make the crc_offset field
larger. We can then use it when finding the offset into the file, since
it will be generic over both v2 and v3 indices.
A v3 index may contain data for only one hash algorithm, so just fail if
we look up an object ID in an algorithm for which the index has no data.
In the pack revindex, we use the order map for the algorithm in question
to look up the proper offset into the pack index, since we're iterating
over objects in the offset table, which is sorted by object ID, not pack
order.
Let's also update t5302 to generate this new format and disable the old
formats when we're using a compatibility hash algorithm, since the old
formats can't represent the SHA-1 values.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Christian Couder <christian.couder@gmail.com>
---
builtin/index-pack.c | 2 +-
hash-lookup.c | 47 +++++++++++++----
hash-lookup.h | 19 ++++++-
pack-check.c | 4 +-
pack-revindex.c | 21 ++++++--
packfile.c | 117 +++++++++++++++++++++++++++++++++++++-----
packfile.h | 18 +++++++
t/t5302-pack-index.sh | 71 +++++++++++++------------
8 files changed, 238 insertions(+), 61 deletions(-)
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 723595ae03..4db3f18951 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1841,7 +1841,7 @@ static void read_idx_option(struct pack_idx_option *opts, const char *pack_name)
/* Read the attributes from the existing idx file */
opts->version = p->index_version;
- if (opts->version == 2)
+ if (opts->version >= 2)
read_v2_anomalous_offsets(p, opts);
/*
diff --git a/hash-lookup.c b/hash-lookup.c
index 5f808caa51..a77ccad8b5 100644
--- a/hash-lookup.c
+++ b/hash-lookup.c
@@ -104,18 +104,13 @@ int oid_pos(const struct object_id *oid, const void *table, size_t nr,
return index_pos_to_insert_pos(lo);
}
-int bsearch_hash(const unsigned char *hash, const uint32_t *fanout_nbo,
- const unsigned char *table, size_t stride, uint32_t *result)
+static int bsearch_hash_1(const unsigned char *hash, const unsigned char *table,
+ size_t stride, size_t cmpsize, uint32_t hi,
+ uint32_t lo, uint32_t *result)
{
- uint32_t hi, lo;
-
- hi = ntohl(fanout_nbo[*hash]);
- lo = ((*hash == 0x0) ? 0 : ntohl(fanout_nbo[*hash - 1]));
-
while (lo < hi) {
unsigned mi = lo + (hi - lo) / 2;
- int cmp = hashcmp(table + mi * stride, hash,
- the_repository->hash_algo);
+ int cmp = memcmp(table + mi * stride, hash, cmpsize);
if (!cmp) {
if (result)
@@ -132,3 +127,37 @@ int bsearch_hash(const unsigned char *hash, const uint32_t *fanout_nbo,
*result = lo;
return 0;
}
+
+int bsearch_hash(const unsigned char *hash, const uint32_t *fanout_nbo,
+ const unsigned char *table, size_t stride, uint32_t *result)
+{
+ uint32_t hi, lo;
+
+ hi = ntohl(fanout_nbo[*hash]);
+ lo = ((*hash == 0x0) ? 0 : ntohl(fanout_nbo[*hash - 1]));
+
+ return bsearch_hash_1(hash, table, stride, the_hash_algo->rawsz, hi, lo, result);
+}
+
+/* full_oid is assumed to follow directly behind short_oid in memory. */
+int bsearch_hash_v3(const unsigned char *hash, const unsigned char *full_oid,
+ const uint32_t *pack_map, const unsigned char *short_oid,
+ size_t stride, size_t hashsz, uint32_t *result)
+{
+ uint32_t obj_off, index;
+ int res = bsearch_hash_1(hash, short_oid, stride, stride, (full_oid - short_oid) / stride,
+ 0, &obj_off);
+ if (result)
+ *result = obj_off;
+ if (!res)
+ return 0;
+ /*
+ * obj_off is now the offset in the shortened list of object names that
+ * we've found. We now need to look it up in the mapping of pack order
+ * to find the full object ID. index is the offset in pack order.
+ */
+ index = ntohl(pack_map[obj_off]);
+ if (!memcmp(hash, full_oid + (hashsz * index), hashsz))
+ return 1;
+ return 0;
+}
diff --git a/hash-lookup.h b/hash-lookup.h
index dbd71ebaf7..03a9be4929 100644
--- a/hash-lookup.h
+++ b/hash-lookup.h
@@ -20,7 +20,7 @@ int oid_pos(const struct object_id *oid,
* first byte is less than or equal to i
* - table: a sorted list of hashes with optional extra information in between
* - stride: distance between two consecutive elements in table (should be
- * GIT_MAX_RAWSZ or greater)
+ * GIT_MAX_RAWSZ or less)
* - result: if not NULL, this function stores the element index of the
* position found (if the search is successful) or the index of the least
* element that is greater than hash (if the search is not successful)
@@ -29,4 +29,21 @@ int oid_pos(const struct object_id *oid,
*/
int bsearch_hash(const unsigned char *hash, const uint32_t *fanout_nbo,
const unsigned char *table, size_t stride, uint32_t *result);
+
+/*
+ * Performs a binary search for hash, returning 1 if it is found and 0 if not.
+ *
+ * - hash is the object ID to search for.
+ * - short_oid is a sorted table of equal-length prefixes of the hashes (each of
+ * which is stride bytes in length.
+ * - full_oid, which must follow in memory directly behind short_oid, is a list
+ * of full object IDs in an arbitrary order (usually pack order); each item is
+ * hashsz bytes in length.
+ * - pack_map is a 4-byte mapping of indices of items in short_oid to full_oid.
+ * - result may be NULL; if not, it is the index of the item in the list of
+ * object IDs if found and the least element greater than it otherwise.
+ */
+int bsearch_hash_v3(const unsigned char *hash, const unsigned char *full_oid,
+ const uint32_t *pack_map, const unsigned char *short_oid,
+ size_t stride, size_t hashsz, uint32_t *result);
#endif
diff --git a/pack-check.c b/pack-check.c
index c7275f4b92..f69a137a18 100644
--- a/pack-check.c
+++ b/pack-check.c
@@ -44,7 +44,9 @@ int check_pack_crc(struct packed_git *p, struct pack_window **w_curs,
} while (len);
index_crc = p->index_data;
- index_crc += 2 + 256 + (size_t)p->num_objects * (p->repo->hash_algo->rawsz/4) + nr;
+ if (p->index_version == 3)
+ nr = nth_packed_object_pack_order_algop(p, nr, p->repo->hash_algo);
+ index_crc += (p->crc_offset / 4) + nr;
return data_crc != ntohl(*index_crc);
}
diff --git a/pack-revindex.c b/pack-revindex.c
index 62387ae632..fb81db8033 100644
--- a/pack-revindex.c
+++ b/pack-revindex.c
@@ -138,11 +138,25 @@ static void create_pack_revindex(struct packed_git *p)
const unsigned hashsz = p->repo->hash_algo->rawsz;
ALLOC_ARRAY(p->revindex, num_ent + 1);
- index += 4 * 256;
- if (p->index_version > 1) {
+ if (p->index_version == 3) {
const uint32_t *off_32 =
- (uint32_t *)(index + 8 + (size_t)p->num_objects * (hashsz + 4));
+ (uint32_t *)(index + st_add(p->crc_offset, st_mult(p->num_objects, 4)));
+ const uint32_t *off_64 = off_32 + p->num_objects;
+ for (i = 0; i < num_ent; i++) {
+ const uint32_t pack_order = nth_packed_object_pack_order_algop(p, i, p->repo->hash_algo);
+ const uint32_t off = ntohl(off_32[pack_order]);
+ const uint32_t lowoff = off & 0x7fffffff;
+ if (!(off & 0x80000000)) {
+ p->revindex[i].offset = off;
+ } else {
+ p->revindex[i].offset = get_be64(off_64 + st_mult(lowoff, 2));
+ }
+ p->revindex[i].nr = i;
+ }
+ } else if (p->index_version == 2) {
+ const uint32_t *off_32 =
+ (uint32_t *)(index + st_add(p->crc_offset, st_mult(p->num_objects, 4)));
const uint32_t *off_64 = off_32 + p->num_objects;
for (i = 0; i < num_ent; i++) {
const uint32_t off = ntohl(*off_32++);
@@ -155,6 +169,7 @@ static void create_pack_revindex(struct packed_git *p)
p->revindex[i].nr = i;
}
} else {
+ index += 4 * 256;
for (i = 0; i < num_ent; i++) {
const uint32_t hl = *((uint32_t *)(index + (hashsz + 4) * i));
p->revindex[i].offset = ntohl(hl);
diff --git a/packfile.c b/packfile.c
index b32b4cd75d..95207c0e19 100644
--- a/packfile.c
+++ b/packfile.c
@@ -88,7 +88,7 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)
return -1;
}
idx_size = xsize_t(st.st_size);
- if (idx_size < 4 * 256 + hashsz + hashsz) {
+ if (idx_size < hashsz * 3) {
close(fd);
return error("index file %s is too small", path);
}
@@ -109,14 +109,14 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,
struct pack_idx_header *hdr = idx_map;
uint32_t version, nr, i, *index;
- if (idx_size < 4 * 256 + hashsz + hashsz)
+ if (idx_size < hashsz * 3)
return error("index file %s is too small", path);
if (!idx_map)
return error("empty data");
if (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) {
version = ntohl(hdr->idx_version);
- if (version < 2 || version > 2)
+ if (version < 2 || version > 3)
return error("index file %s is version %"PRIu32
" and is not supported by this binary"
" (try upgrading GIT to a newer version)",
@@ -124,15 +124,20 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,
} else
version = 1;
+ if (version < 3 && idx_size < 4 * 256 + hashsz + hashsz)
+ return error("index file %s is too small", path);
+
nr = 0;
index = idx_map;
if (version > 1)
index += 2; /* skip index header */
- for (i = 0; i < 256; i++) {
- uint32_t n = ntohl(index[i]);
- if (n < nr)
- return error("non-monotonic index %s", path);
- nr = n;
+ if (version <= 2) {
+ for (i = 0; i < 256; i++) {
+ uint32_t n = ntohl(index[i]);
+ if (n < nr)
+ return error("non-monotonic index %s", path);
+ nr = n;
+ }
}
if (version == 1) {
@@ -174,6 +179,51 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,
(sizeof(off_t) <= 4))
return error("pack too large for current definition of off_t in %s", path);
p->crc_offset = st_add(8 + 4 * 256, st_mult(nr, hashsz));
+ } else if (version == 3) {
+ uint32_t hdrlen, num_formats, trailer_offset, main_algo;
+ const uint32_t *q = index;
+ int i;
+
+ hdrlen = ntohl(*q++);
+ if (idx_size < hdrlen)
+ return error("v3 index has truncated header");
+
+ nr = ntohl(*q++);
+ num_formats = ntohl(*q++);
+ if (num_formats < 1 || num_formats > ARRAY_SIZE(p->formats))
+ return error("too many formats for v3 index");
+ memset(p->formats, 0, sizeof(p->formats));
+ for (i = 0; i < num_formats; i++) {
+ uint32_t format_id = ntohl(*q++);
+ uint32_t format = hash_algo_by_id(format_id);
+ struct packed_git_format *pfp;
+
+ if (format == GIT_HASH_UNKNOWN)
+ return error("v3 index with unknown format_id 0x%08x", format_id);
+ /*
+ * We store at the value of the constant minus 1 to
+ * avoid having to store a useless entry at offset 0.
+ */
+ pfp = &p->formats[format-1];
+
+ pfp->short_name_len = ntohl(*q++);
+ pfp->data_offset = get_be64(q);
+ q += 2;
+
+ pfp->full_oid_offset = pfp->data_offset +
+ (pfp->short_name_len * nr);
+ pfp->order_map_offset = pfp->full_oid_offset +
+ (hash_algos[format].rawsz * nr);
+ if (!i)
+ p->crc_offset = pfp->order_map_offset + nr * 4;
+ }
+ main_algo = hash_algo_by_length(hashsz);
+ if (main_algo == GIT_HASH_UNKNOWN ||
+ !p->formats[main_algo - 1].data_offset)
+ return error("v3 index with missing data for this repository's algorithm");
+ trailer_offset = get_be64(q);
+ if (trailer_offset + 2 * hashsz != idx_size)
+ return error("v3 index with bad size\n");
}
p->index_version = version;
@@ -1730,6 +1780,26 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,
return data;
}
+static int bsearch_pack_v3(const struct object_id *oid, const struct packed_git *p, uint32_t *result)
+{
+ int algo = oid->algo ? oid->algo : hash_algo_by_ptr(p->repo->hash_algo);
+ const struct packed_git_format *pfp = &p->formats[algo - 1];
+ const unsigned char *data = p->index_data;
+
+ /* This index has no data for the algorithm of the object ID. */
+ if (!pfp->data_offset) {
+ if (result)
+ *result = 0;
+ return 0;
+ }
+
+ return bsearch_hash_v3(oid->hash, data + pfp->full_oid_offset,
+ (const uint32_t *)(data + pfp->order_map_offset),
+ data + pfp->data_offset,
+ pfp->short_name_len,
+ hash_algos[algo].rawsz, result);
+}
+
int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32_t *result)
{
const unsigned char *index_fanout = p->index_data;
@@ -1740,6 +1810,9 @@ int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32
if (!index_fanout)
BUG("bsearch_pack called without a valid pack-index");
+ if (p->index_version == 3)
+ return bsearch_pack_v3(oid, p, result);
+
index_lookup = index_fanout + 4 * 256;
if (p->index_version == 1) {
index_lookup_width = hashsz + 4;
@@ -1767,13 +1840,17 @@ int nth_packed_object_id(struct object_id *oid,
}
if (n >= p->num_objects)
return -1;
- index += 4 * 256;
if (p->index_version == 1) {
+ index += 4 * 256;
oidread(oid, index + st_add(st_mult(hashsz + 4, n), 4),
p->repo->hash_algo);
- } else {
- index += 8;
+ } else if (p->index_version == 2) {
+ index += 4 * 256 + 8;
oidread(oid, index + st_mult(hashsz, n), p->repo->hash_algo);
+ } else {
+ int algo = hash_algo_by_ptr(p->repo->hash_algo);
+ const struct packed_git_format *pfp = &p->formats[algo-1];
+ oidread(oid, index + pfp->full_oid_offset + (hashsz * n), p->repo->hash_algo);
}
return 0;
}
@@ -1796,12 +1873,14 @@ off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)
{
const unsigned char *index = p->index_data;
const unsigned int hashsz = p->repo->hash_algo->rawsz;
- index += 4 * 256;
if (p->index_version == 1) {
+ index += 4 * 256;
return ntohl(*((uint32_t *)(index + st_mult(hashsz + 4, n))));
} else {
uint32_t off;
- index += st_add(8, st_mult(p->num_objects, hashsz + 4));
+ if (p->index_version == 3)
+ n = nth_packed_object_pack_order_algop(p, n, p->repo->hash_algo);
+ index += st_add(p->crc_offset, st_mult(p->num_objects, 4));
off = ntohl(*((uint32_t *)(index + st_mult(4, n))));
if (!(off & 0x80000000))
return off;
@@ -1812,6 +1891,18 @@ off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)
}
}
+uint32_t nth_packed_object_pack_order_algop(const struct packed_git *p, uint32_t n,
+ const struct git_hash_algo *algop)
+{
+ const unsigned char *index = p->index_data;
+ const unsigned int algo = hash_algo_by_ptr(algop);
+ const struct packed_git_format *pfp = &p->formats[algo-1];
+
+ return ntohl(*(uint32_t *)(index +
+ st_add(pfp->order_map_offset,
+ st_mult(4, n))));
+}
+
off_t find_pack_entry_one(const struct object_id *oid,
struct packed_git *p)
{
diff --git a/packfile.h b/packfile.h
index ed3378c457..22278cc6c6 100644
--- a/packfile.h
+++ b/packfile.h
@@ -1,6 +1,7 @@
#ifndef PACKFILE_H
#define PACKFILE_H
+#include "hash.h"
#include "list.h"
#include "object.h"
#include "odb.h"
@@ -14,12 +15,22 @@
struct object_info;
struct odb_stream;
+struct packed_git_format {
+ /* This is also the location of the short OID table. */
+ uint64_t data_offset;
+ uint64_t full_oid_offset;
+ uint64_t order_map_offset;
+ uint32_t short_name_len;
+};
+
struct packed_git {
struct pack_window *windows;
off_t pack_size;
const void *index_data;
size_t index_size;
uint32_t num_objects;
+ uint32_t num_formats;
+ struct packed_git_format formats[GIT_HASH_NALGOS - 1];
size_t crc_offset;
struct oidset bad_objects;
int index_version;
@@ -284,6 +295,13 @@ int nth_packed_object_id(struct object_id *, struct packed_git *, uint32_t n);
*/
off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);
+/*
+ * Return the index into pack order of the nth object within the specified
+ * packfile. The index must already be opened and it must be a v3 index.
+ */
+uint32_t nth_packed_object_pack_order_algop(const struct packed_git *, uint32_t n,
+ const struct git_hash_algo *algop);
+
/*
* If the object named by oid is present in the specified packfile,
* return its offset within the packfile; otherwise, return 0.
diff --git a/t/t5302-pack-index.sh b/t/t5302-pack-index.sh
index 735de1023e..9ea0ed487e 100755
--- a/t/t5302-pack-index.sh
+++ b/t/t5302-pack-index.sh
@@ -36,50 +36,55 @@ test_expect_success 'setup' '
git update-ref HEAD $commit
'
-test_expect_success 'pack-objects with index version 1' '
+test_expect_success !COMPAT_HASH 'pack-objects with index version 1' '
pack1=$(git pack-objects --index-version=1 test-1 <obj-list) &&
git verify-pack -v "test-1-${pack1}.pack"
'
-test_expect_success 'pack-objects with index version 2' '
+test_expect_success !COMPAT_HASH 'pack-objects with index version 2' '
pack2=$(git pack-objects --index-version=2 test-2 <obj-list) &&
git verify-pack -v "test-2-${pack2}.pack"
'
-test_expect_success 'both packs should be identical' '
+test_expect_success 'pack-objects with index version 3' '
+ packv3=$(git pack-objects --index-version=3 test-v3 <obj-list) &&
+ git verify-pack -v "test-v3-${packv3}.pack"
+'
+
+test_expect_success !COMPAT_HASH 'both packs should be identical' '
cmp "test-1-${pack1}.pack" "test-2-${pack2}.pack"
'
-test_expect_success 'index v1 and index v2 should be different' '
+test_expect_success !COMPAT_HASH 'index v1 and index v2 should be different' '
! cmp "test-1-${pack1}.idx" "test-2-${pack2}.idx"
'
-test_expect_success 'index-pack with index version 1' '
+test_expect_success !COMPAT_HASH 'index-pack with index version 1' '
git index-pack --index-version=1 -o 1.idx "test-1-${pack1}.pack"
'
-test_expect_success 'index-pack with index version 2' '
+test_expect_success !COMPAT_HASH 'index-pack with index version 2' '
git index-pack --index-version=2 -o 2.idx "test-1-${pack1}.pack"
'
-test_expect_success 'index-pack results should match pack-objects ones' '
+test_expect_success !COMPAT_HASH 'index-pack results should match pack-objects ones' '
cmp "test-1-${pack1}.idx" "1.idx" &&
cmp "test-2-${pack2}.idx" "2.idx"
'
-test_expect_success 'index-pack --verify on index version 1' '
+test_expect_success !COMPAT_HASH 'index-pack --verify on index version 1' '
git index-pack --verify "test-1-${pack1}.pack"
'
-test_expect_success 'index-pack --verify on index version 2' '
+test_expect_success !COMPAT_HASH 'index-pack --verify on index version 2' '
git index-pack --verify "test-2-${pack2}.pack"
'
-test_expect_success 'pack-objects --index-version=2, is not accepted' '
+test_expect_success !COMPAT_HASH 'pack-objects --index-version=2, is not accepted' '
test_must_fail git pack-objects --index-version=2, test-3 <obj-list
'
-test_expect_success 'index v2: force some 64-bit offsets with pack-objects' '
+test_expect_success !COMPAT_HASH 'index v2: force some 64-bit offsets with pack-objects' '
pack3=$(git pack-objects --index-version=2,0x40000 test-3 <obj-list)
'
@@ -91,29 +96,29 @@ else
say "# skipping tests concerning 64-bit offsets"
fi
-test_expect_success OFF64_T 'index v2: verify a pack with some 64-bit offsets' '
+test_expect_success OFF64_T,!COMPAT_HASH 'index v2: verify a pack with some 64-bit offsets' '
git verify-pack -v "test-3-${pack3}.pack"
'
-test_expect_success OFF64_T '64-bit offsets: should be different from previous index v2 results' '
+test_expect_success OFF64_T,!COMPAT_HASH '64-bit offsets: should be different from previous index v2 results' '
! cmp "test-2-${pack2}.idx" "test-3-${pack3}.idx"
'
-test_expect_success OFF64_T 'index v2: force some 64-bit offsets with index-pack' '
+test_expect_success OFF64_T,!COMPAT_HASH 'index v2: force some 64-bit offsets with index-pack' '
git index-pack --index-version=2,0x40000 -o 3.idx "test-1-${pack1}.pack"
'
-test_expect_success OFF64_T '64-bit offsets: index-pack result should match pack-objects one' '
+test_expect_success OFF64_T,!COMPAT_HASH '64-bit offsets: index-pack result should match pack-objects one' '
cmp "test-3-${pack3}.idx" "3.idx"
'
-test_expect_success OFF64_T 'index-pack --verify on 64-bit offset v2 (cheat)' '
+test_expect_success OFF64_T,!COMPAT_HASH 'index-pack --verify on 64-bit offset v2 (cheat)' '
# This cheats by knowing which lower offset should still be encoded
# in 64-bit representation.
git index-pack --verify --index-version=2,0x40000 "test-3-${pack3}.pack"
'
-test_expect_success OFF64_T 'index-pack --verify on 64-bit offset v2' '
+test_expect_success OFF64_T,!COMPAT_HASH 'index-pack --verify on 64-bit offset v2' '
git index-pack --verify "test-3-${pack3}.pack"
'
@@ -142,7 +147,7 @@ index_obj_offset()
( read offs extra && echo "$offs" )
}
-test_expect_success '[index v1] 1) stream pack to repository' '
+test_expect_success !COMPAT_HASH '[index v1] 1) stream pack to repository' '
git index-pack --index-version=1 --stdin < "test-1-${pack1}.pack" &&
git prune-packed &&
git count-objects | ( read nr rest && test "$nr" -eq 1 ) &&
@@ -150,7 +155,7 @@ test_expect_success '[index v1] 1) stream pack to repository' '
cmp "test-1-${pack1}.idx" ".git/objects/pack/pack-${pack1}.idx"
'
-test_expect_success \
+test_expect_success !COMPAT_HASH \
'[index v1] 2) create a stealth corruption in a delta base reference' '
# This test assumes file_101 is a delta smaller than 16 bytes.
# It should be against file_100 but we substitute its base for file_099
@@ -167,27 +172,27 @@ test_expect_success \
git cat-file blob $sha1_101 > file_101_foo1
'
-test_expect_success \
+test_expect_success !COMPAT_HASH \
'[index v1] 3) corrupted delta happily returned wrong data' '
test -f file_101_foo1 && ! cmp file_101 file_101_foo1
'
-test_expect_success \
+test_expect_success !COMPAT_HASH \
'[index v1] 4) confirm that the pack is actually corrupted' '
test_must_fail git fsck --full $commit
'
-test_expect_success \
+test_expect_success !COMPAT_HASH \
'[index v1] 5) pack-objects happily reuses corrupted data' '
pack4=$(git pack-objects test-4 <obj-list) &&
test -f "test-4-${pack4}.pack"
'
-test_expect_success '[index v1] 6) newly created pack is BAD !' '
+test_expect_success !COMPAT_HASH '[index v1] 6) newly created pack is BAD !' '
test_must_fail git verify-pack -v "test-4-${pack4}.pack"
'
-test_expect_success '[index v2] 1) stream pack to repository' '
+test_expect_success !COMPAT_HASH '[index v2] 1) stream pack to repository' '
rm -f .git/objects/pack/* &&
git index-pack --index-version=2 --stdin < "test-1-${pack1}.pack" &&
git prune-packed &&
@@ -196,7 +201,7 @@ test_expect_success '[index v2] 1) stream pack to repository' '
cmp "test-2-${pack1}.idx" ".git/objects/pack/pack-${pack1}.idx"
'
-test_expect_success \
+test_expect_success !COMPAT_HASH \
'[index v2] 2) create a stealth corruption in a delta base reference' '
# This test assumes file_101 is a delta smaller than 16 bytes.
# It should be against file_100 but we substitute its base for file_099
@@ -212,23 +217,23 @@ test_expect_success \
git cat-file blob $sha1_101 > file_101_foo2
'
-test_expect_success \
+test_expect_success !COMPAT_HASH \
'[index v2] 3) corrupted delta happily returned wrong data' '
test -f file_101_foo2 && ! cmp file_101 file_101_foo2
'
-test_expect_success \
+test_expect_success !COMPAT_HASH \
'[index v2] 4) confirm that the pack is actually corrupted' '
test_must_fail git fsck --full $commit
'
-test_expect_success \
+test_expect_success !COMPAT_HASH \
'[index v2] 5) pack-objects refuses to reuse corrupted data' '
test_must_fail git pack-objects test-5 <obj-list &&
test_must_fail git pack-objects --no-reuse-object test-6 <obj-list
'
-test_expect_success \
+test_expect_success !COMPAT_HASH \
'[index v2] 6) verify-pack detects CRC mismatch' '
rm -f .git/objects/pack/* &&
git index-pack --index-version=2 --stdin < "test-1-${pack1}.pack" &&
@@ -246,12 +251,12 @@ test_expect_success \
test_expect_success 'running index-pack in the object store' '
rm -f .git/objects/pack/* &&
- cp test-1-${pack1}.pack .git/objects/pack/pack-${pack1}.pack &&
+ cp test-v3-${packv3}.pack .git/objects/pack/pack-${packv3}.pack &&
(
cd .git/objects/pack &&
- git index-pack pack-${pack1}.pack
+ git index-pack pack-${packv3}.pack
) &&
- test -f .git/objects/pack/pack-${pack1}.idx
+ test -f .git/objects/pack/pack-${packv3}.idx
'
test_expect_success 'index-pack --strict warns upon missing tagger in tag' '
@@ -273,7 +278,7 @@ EOF
test_grep "^warning:.* expected .tagger. line" err
'
-test_expect_success 'index-pack --fsck-objects also warns upon missing tagger in tag' '
+test_expect_success !COMPAT_HASH 'index-pack --fsck-objects also warns upon missing tagger in tag' '
git index-pack --fsck-objects tag-test-${pack1}.pack 2>err &&
test_grep "^warning:.* expected .tagger. line" err
'
--
2.56.0.215.g47fdf2208c
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 10/13] packfile: validate pack index v3 tables
2026-10-23 7:25 [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Christian Couder
` (8 preceding siblings ...)
2026-10-23 7:25 ` [PATCH v2 09/13] packfile: add support for reading pack index v3 Christian Couder
@ 2026-10-23 7:25 ` Christian Couder
2026-10-23 7:25 ` [PATCH v2 11/13] loose: map object IDs using pack index v3 Christian Couder
` (4 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Christian Couder @ 2026-10-23 7:25 UTC (permalink / raw)
To: git
Cc: Junio C Hamano, brian m . carlson, Patrick Steinhardt,
Elijah Newren, Jeff King, Johannes Schindelin, Christian Couder
A previous commit implemented loading a pack index v3, but didn't
check much of its format when doing so.
To avoid reading outside a v3 index, if it's corrupt or crafted, let's
check that its header, its format entries and all its tables lie within
the file, when loading it.
Let's also reject duplicate formats and short names longer than the
full object names.
The values in the order maps are checked when they are read, rather
than when loading the index, so that loading stays cheap. A new test in
t5313 ensures that an invalid order map entry is detected.
In the pack revindex, let's also check that the 8-byte offsets we read
lie within the index.
Signed-off-by: Christian Couder <christian.couder@gmail.com>
---
hash-lookup.c | 5 ++-
pack-revindex.c | 5 ++-
packfile.c | 71 ++++++++++++++++++++++++++---------
t/t5313-pack-bounds-checks.sh | 17 +++++++++
4 files changed, 78 insertions(+), 20 deletions(-)
diff --git a/hash-lookup.c b/hash-lookup.c
index a77ccad8b5..3f0725f7ec 100644
--- a/hash-lookup.c
+++ b/hash-lookup.c
@@ -144,8 +144,9 @@ int bsearch_hash_v3(const unsigned char *hash, const unsigned char *full_oid,
const uint32_t *pack_map, const unsigned char *short_oid,
size_t stride, size_t hashsz, uint32_t *result)
{
+ uint32_t nr = (full_oid - short_oid) / stride;
uint32_t obj_off, index;
- int res = bsearch_hash_1(hash, short_oid, stride, stride, (full_oid - short_oid) / stride,
+ int res = bsearch_hash_1(hash, short_oid, stride, stride, nr,
0, &obj_off);
if (result)
*result = obj_off;
@@ -157,6 +158,8 @@ int bsearch_hash_v3(const unsigned char *hash, const unsigned char *full_oid,
* to find the full object ID. index is the offset in pack order.
*/
index = ntohl(pack_map[obj_off]);
+ if (index >= nr)
+ return 0;
if (!memcmp(hash, full_oid + (hashsz * index), hashsz))
return 1;
return 0;
diff --git a/pack-revindex.c b/pack-revindex.c
index fb81db8033..4e6923cfd3 100644
--- a/pack-revindex.c
+++ b/pack-revindex.c
@@ -150,7 +150,10 @@ static void create_pack_revindex(struct packed_git *p)
if (!(off & 0x80000000)) {
p->revindex[i].offset = off;
} else {
- p->revindex[i].offset = get_be64(off_64 + st_mult(lowoff, 2));
+ const uint32_t *ptr = off_64 + st_mult(lowoff, 2);
+
+ check_pack_index_ptr(p, ptr);
+ p->revindex[i].offset = get_be64(ptr);
}
p->revindex[i].nr = i;
}
diff --git a/packfile.c b/packfile.c
index 95207c0e19..f6e2db45d5 100644
--- a/packfile.c
+++ b/packfile.c
@@ -180,50 +180,81 @@ int load_idx(const char *path, const unsigned int hashsz, void *idx_map,
return error("pack too large for current definition of off_t in %s", path);
p->crc_offset = st_add(8 + 4 * 256, st_mult(nr, hashsz));
} else if (version == 3) {
- uint32_t hdrlen, num_formats, trailer_offset, main_algo;
+ uint32_t hdrlen, num_formats, main_algo;
+ uint64_t trailer_offset;
const uint32_t *q = index;
- int i;
+ /* Fixed part of the header (after signature and version) */
hdrlen = ntohl(*q++);
- if (idx_size < hdrlen)
- return error("v3 index has truncated header");
-
nr = ntohl(*q++);
num_formats = ntohl(*q++);
if (num_formats < 1 || num_formats > ARRAY_SIZE(p->formats))
- return error("too many formats for v3 index");
+ return error("v3 index %s has an invalid number of formats: %"PRIu32,
+ path, num_formats);
+
+ /* Then 16 bytes per format and the 8-byte offset of the trailer */
+ if (hdrlen < 20 + num_formats * (4 + 4 + 8) + 8 ||
+ hdrlen > idx_size - 2 * hashsz)
+ return error("v3 index %s has an invalid header length", path);
+
+ trailer_offset = get_be64(q + num_formats * 4);
+ if (trailer_offset != idx_size - 2 * hashsz)
+ return error("v3 index %s has a bad size", path);
+
memset(p->formats, 0, sizeof(p->formats));
for (i = 0; i < num_formats; i++) {
uint32_t format_id = ntohl(*q++);
uint32_t format = hash_algo_by_id(format_id);
struct packed_git_format *pfp;
+ uint64_t end;
if (format == GIT_HASH_UNKNOWN)
- return error("v3 index with unknown format_id 0x%08x", format_id);
+ return error("v3 index %s has an unknown format_id 0x%08"PRIx32,
+ path, format_id);
/*
* We store at the value of the constant minus 1 to
* avoid having to store a useless entry at offset 0.
*/
pfp = &p->formats[format-1];
+ if (pfp->data_offset)
+ return error("v3 index %s has duplicate format_id 0x%08"PRIx32,
+ path, format_id);
pfp->short_name_len = ntohl(*q++);
pfp->data_offset = get_be64(q);
q += 2;
+ if (!pfp->short_name_len ||
+ pfp->short_name_len > hash_algos[format].rawsz)
+ return error("v3 index %s has an invalid short name length",
+ path);
+
+ if (pfp->data_offset < hdrlen ||
+ pfp->data_offset > trailer_offset)
+ return error("v3 index %s has an invalid table offset", path);
+
+ /* No overflow as pfp->data_offset < idx_size and nr < 2^32 */
pfp->full_oid_offset = pfp->data_offset +
- (pfp->short_name_len * nr);
+ (uint64_t)pfp->short_name_len * nr;
pfp->order_map_offset = pfp->full_oid_offset +
- (hash_algos[format].rawsz * nr);
- if (!i)
- p->crc_offset = pfp->order_map_offset + nr * 4;
+ (uint64_t)hash_algos[format].rawsz * nr;
+ end = pfp->order_map_offset + (uint64_t)nr * 4;
+
+ /* After the first format tables: CRC32 and 4-byte offset tables */
+ if (!i) {
+ p->crc_offset = end;
+ end += (uint64_t)nr * (4 + 4); /* CRC32 + offset */
+ }
+
+ if (pfp->order_map_offset % 4 || end > trailer_offset)
+ return error("v3 index %s has invalid tables", path);
}
+
main_algo = hash_algo_by_length(hashsz);
if (main_algo == GIT_HASH_UNKNOWN ||
!p->formats[main_algo - 1].data_offset)
- return error("v3 index with missing data for this repository's algorithm");
- trailer_offset = get_be64(q);
- if (trailer_offset + 2 * hashsz != idx_size)
- return error("v3 index with bad size\n");
+ return error("v3 index %s is missing data for the main hash algorithm",
+ path);
}
p->index_version = version;
@@ -1897,10 +1928,14 @@ uint32_t nth_packed_object_pack_order_algop(const struct packed_git *p, uint32_t
const unsigned char *index = p->index_data;
const unsigned int algo = hash_algo_by_ptr(algop);
const struct packed_git_format *pfp = &p->formats[algo-1];
+ uint32_t pos = ntohl(*(uint32_t *)(index +
+ st_add(pfp->order_map_offset,
+ st_mult(4, n))));
- return ntohl(*(uint32_t *)(index +
- st_add(pfp->order_map_offset,
- st_mult(4, n))));
+ if (pos >= p->num_objects)
+ die(_("invalid object position in pack index for %s (corrupt index?)"),
+ p->pack_name);
+ return pos;
}
off_t find_pack_entry_one(const struct object_id *oid,
diff --git a/t/t5313-pack-bounds-checks.sh b/t/t5313-pack-bounds-checks.sh
index 5be01260d7..ab2b0bb9c6 100755
--- a/t/t5313-pack-bounds-checks.sh
+++ b/t/t5313-pack-bounds-checks.sh
@@ -45,6 +45,14 @@ extended_table () {
echo $(($(ofs_table "$1") + 4*$1))
}
+# Offset in a single-format v3 .idx with no extended offsets to the
+# start of its order map. $1 is its number of objects, nr.
+# The order map (4*nr) is followed by the CRC32 table (4*nr), the
+# 4-byte offset table (4*nr) and by the trailer (2*rawsz).
+v3_order_map () {
+ echo $(($(wc -c <"$idx") - 2*$(test_oid rawsz) - 12*$1))
+}
+
test_expect_success 'setup' '
test_oid_cache <<-EOF
oid000 sha1:1485
@@ -171,6 +179,15 @@ test_expect_success 'bogus offset inside v2 extended table' '
test_must_fail git index-pack --verify $pack
'
+test_expect_success 'bogus order map entry (v3)' '
+ do_pack $object --index-version=3 &&
+ munge $idx $(v3_order_map 1) "\377\377\377\377" &&
+ clear_base &&
+ test_must_fail git cat-file blob $object &&
+ test_must_fail git cat-file --batch-all-objects --batch-check 2>err &&
+ test_grep "corrupt index" err
+'
+
test_expect_success 'bogus OFS_DELTA in packfile' '
# Generate a pack with a delta in it.
base=$(test-tool genrandom foo 3000 | git hash-object --stdin -w) &&
--
2.56.0.215.g47fdf2208c
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 11/13] loose: map object IDs using pack index v3
2026-10-23 7:25 [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Christian Couder
` (9 preceding siblings ...)
2026-10-23 7:25 ` [PATCH v2 10/13] packfile: validate pack index v3 tables Christian Couder
@ 2026-10-23 7:25 ` Christian Couder
2026-10-23 7:25 ` [PATCH v2 12/13] packfile: allow looking up compat object IDs in packs Christian Couder
` (3 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Christian Couder @ 2026-10-23 7:25 UTC (permalink / raw)
To: git
Cc: Junio C Hamano, brian m . carlson, Patrick Steinhardt,
Elijah Newren, Jeff King, Johannes Schindelin, Christian Couder
From: "brian m. carlson" <sandals@crustytoothpaste.net>
Right now, we can only map an object if it exists in the loose object
map. However, we have a perfectly good way to look up packed objects,
which is the pack index. Let's use that to look up any packed objects
if they're not in the loose object map.
In order to do so, let's add a variant of nth_packed_object_id that
handles an arbitrary hash algorithm.
Only pack index v3 stores object IDs in more than one algorithm, so
skip packs with an older index when mapping.
Also make the new variant fail when the index cannot provide the
requested object ID, for example because it has no data for one of the
algorithms.
Add a test in t1016 to check that packed objects are mapped using a v3
index, but not using a v2 index, when the loose object map has no entry
for them.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Christian Couder <christian.couder@gmail.com>
---
odb/source-packed.c | 30 ++++++++++++++++++++++++++----
packfile.c | 28 +++++++++++++++++++++++-----
packfile.h | 8 ++++++++
t/t1016-compatObjectFormat.sh | 27 +++++++++++++++++++++++++++
4 files changed, 84 insertions(+), 9 deletions(-)
diff --git a/odb/source-packed.c b/odb/source-packed.c
index ee1edfdb2b..4ecc2de818 100644
--- a/odb/source-packed.c
+++ b/odb/source-packed.c
@@ -863,12 +863,34 @@ static void odb_source_packed_close(struct odb_source *source)
packed->midx = NULL;
}
-static int odb_source_packed_map_oid(struct odb_source *source UNUSED,
- const struct object_id *src UNUSED,
- const struct git_hash_algo *to UNUSED,
- struct object_id *dest UNUSED,
+static int odb_source_packed_map_oid(struct odb_source *source,
+ const struct object_id *src,
+ const struct git_hash_algo *to,
+ struct object_id *dest,
enum odb_map_oid_flags flags UNUSED)
{
+ struct odb_source_packed *packed = odb_source_packed_downcast(source);
+ struct repository *repo = packed->base.odb->repo;
+ struct packfile_list_entry *e;
+
+ for (e = packfile_store_get_packs(packed); e; e = e->next) {
+ struct packed_git *p = e->pack;
+ uint32_t off;
+ const struct git_hash_algo *src_algo = src->algo ?
+ &hash_algos[src->algo] :
+ repo->hash_algo;
+
+ if (open_pack_index(p) || !p->num_objects)
+ continue;
+ /* Only v3 indices store object IDs in both algorithms. */
+ if (p->index_version < 3)
+ continue;
+ if (!bsearch_pack(src, p, &off))
+ continue;
+ if (nth_packed_object_id_algop(dest, p, off, src_algo, to))
+ continue;
+ return 0;
+ }
return -1;
}
diff --git a/packfile.c b/packfile.c
index f6e2db45d5..3bfd427c6c 100644
--- a/packfile.c
+++ b/packfile.c
@@ -1861,9 +1861,19 @@ int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32
int nth_packed_object_id(struct object_id *oid,
struct packed_git *p,
uint32_t n)
+{
+ return nth_packed_object_id_algop(oid, p, n, p->repo->hash_algo,
+ p->repo->hash_algo);
+}
+
+int nth_packed_object_id_algop(struct object_id *oid,
+ struct packed_git *p,
+ uint32_t n,
+ const struct git_hash_algo *src,
+ const struct git_hash_algo *dest)
{
const unsigned char *index = p->index_data;
- const unsigned int hashsz = p->repo->hash_algo->rawsz;
+ const unsigned int hashsz = dest->rawsz;
if (!index) {
if (open_pack_index(p))
return -1;
@@ -1871,17 +1881,25 @@ int nth_packed_object_id(struct object_id *oid,
}
if (n >= p->num_objects)
return -1;
+ if (p->index_version < 3 && (src != dest || dest != p->repo->hash_algo))
+ return -1; /* older indices only store main object IDs */
if (p->index_version == 1) {
index += 4 * 256;
oidread(oid, index + st_add(st_mult(hashsz + 4, n), 4),
- p->repo->hash_algo);
+ dest);
} else if (p->index_version == 2) {
index += 4 * 256 + 8;
- oidread(oid, index + st_mult(hashsz, n), p->repo->hash_algo);
+ oidread(oid, index + st_mult(hashsz, n), dest);
} else {
- int algo = hash_algo_by_ptr(p->repo->hash_algo);
+ int algo = hash_algo_by_ptr(dest);
const struct packed_git_format *pfp = &p->formats[algo-1];
- oidread(oid, index + pfp->full_oid_offset + (hashsz * n), p->repo->hash_algo);
+ uint32_t pack_order_idx;
+
+ if (!pfp->data_offset ||
+ !p->formats[hash_algo_by_ptr(src) - 1].data_offset)
+ return -1; /* no data for one of the algorithms */
+ pack_order_idx = nth_packed_object_pack_order_algop(p, n, src);
+ oidread(oid, index + st_add(pfp->full_oid_offset, st_mult(hashsz, pack_order_idx)), dest);
}
return 0;
}
diff --git a/packfile.h b/packfile.h
index 22278cc6c6..73452b2a43 100644
--- a/packfile.h
+++ b/packfile.h
@@ -289,6 +289,14 @@ int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32
*/
int nth_packed_object_id(struct object_id *, struct packed_git *, uint32_t n);
+/*
+ * The same as nth_packed_object_id, but the nth object is counted in src
+ * and the object ID looked up is in dest.
+ */
+int nth_packed_object_id_algop(struct object_id *, struct packed_git *,
+ uint32_t n, const struct git_hash_algo *src,
+ const struct git_hash_algo *dest);
+
/*
* Return the offset of the nth object within the specified packfile.
* The index must already be opened.
diff --git a/t/t1016-compatObjectFormat.sh b/t/t1016-compatObjectFormat.sh
index af9532aa1f..030d6000b5 100755
--- a/t/t1016-compatObjectFormat.sh
+++ b/t/t1016-compatObjectFormat.sh
@@ -242,6 +242,33 @@ do
git -C large-repo cat-file blob :large >actual &&
test_cmp large-repo/large actual
'
+
+ test_expect_success 'packed objects are only mapped using pack index v3' '
+ test_when_finished "rm -rf map-repo v3.idx loose-object-idx" &&
+ create_compat_repo $hash map-repo &&
+ echo content >map-repo/file &&
+ git -C map-repo add file &&
+ git -C map-repo commit --message A &&
+ oid=$(git -C map-repo rev-parse HEAD) &&
+ git -C map-repo rev-parse --output-object-format=$(compat_hash $hash) "$oid" >expect &&
+ git -C map-repo repack -a -d &&
+ pack=$(echo map-repo/.git/objects/pack/pack-*.pack) &&
+ git -C map-repo index-pack --index-version=3 \
+ -o "$(pwd)/v3.idx" "$(pwd)/$pack" &&
+ mv map-repo/.git/objects/loose-object-idx loose-object-idx &&
+
+ # A v2 index only has object IDs in the main algorithm, so it
+ # must not provide a mapping.
+ git -C map-repo rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
+ ! test_cmp expect actual &&
+ test $(wc -c <actual) -ne $(($(test_oid --hash=$(compat_hash $hash) hexsz) + 1)) &&
+
+ # A v3 index provides it.
+ rm -f "${pack%.pack}.idx" "${pack%.pack}.rev" &&
+ mv v3.idx "${pack%.pack}.idx" &&
+ git -C map-repo rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
+ test_cmp expect actual
+ '
done
cd "$base"
--
2.56.0.215.g47fdf2208c
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 12/13] packfile: allow looking up compat object IDs in packs
2026-10-23 7:25 [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Christian Couder
` (10 preceding siblings ...)
2026-10-23 7:25 ` [PATCH v2 11/13] loose: map object IDs using pack index v3 Christian Couder
@ 2026-10-23 7:25 ` Christian Couder
2026-10-23 7:25 ` [PATCH v2 13/13] doc: move pack index v3 documentation Christian Couder
` (2 subsequent siblings)
14 siblings, 0 replies; 17+ messages in thread
From: Christian Couder @ 2026-10-23 7:25 UTC (permalink / raw)
To: git
Cc: Junio C Hamano, brian m . carlson, Patrick Steinhardt,
Elijah Newren, Jeff King, Johannes Schindelin, Christian Couder
From: "brian m. carlson" <sandals@crustytoothpaste.net>
Now that we have the ability to store compatibility object IDs in pack
index v3, we also want to be able to look up these object IDs. Our
disambiguation code can work with a particular hash algorithm or with
any hash algorithm, the most notable instance of the latter case being
when git rev-parse is invoked with --output-object-format.
We already handle the single algorithm case just fine, but we don't
gracefully handle resolving an object ID in any algorithm. Solve this
by iterating through the algorithms and, if the algorithm is both one of
the desired ones and one our repository supports, then try looking up
the object IDs for that algorithm. Use an intermediate variable so that
we can pass the correct algorithm in the object ID's algo member, which
will cause the correct tables to be looked up.
Write this check as two if statements because the logic is complex and
it is easier to understand as two statements. The compiler will
optimize it appropriately.
When looking up an object ID, we find positions in the table sorted by
that object ID's algorithm. So read the object IDs there for that
algorithm with nth_packed_object_id_algop(), and add
nth_packed_object_offset_algop() to get their offsets.
Skip the compatibility algorithm for packs with an index older than v3,
as these only store object IDs in the main algorithm.
Add a test that looks up an abbreviated compatibility object ID of a
packed object.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Christian Couder <christian.couder@gmail.com>
---
odb/source-packed.c | 83 +++++++++++++++++++++++------------
packfile.c | 13 +++++-
packfile.h | 7 +++
t/t1016-compatObjectFormat.sh | 30 +++++++++++++
4 files changed, 104 insertions(+), 29 deletions(-)
diff --git a/odb/source-packed.c b/odb/source-packed.c
index 4ecc2de818..8b16e8da29 100644
--- a/odb/source-packed.c
+++ b/odb/source-packed.c
@@ -295,37 +295,64 @@ static int for_each_prefixed_object_in_pack(
int len = opts->prefix_hex_len > p->repo->hash_algo->hexsz ?
p->repo->hash_algo->hexsz : opts->prefix_hex_len;
int ret;
+ struct object_id wanted;
num = p->num_objects;
- bsearch_pack(opts->prefix, p, &first);
- /*
- * At this point, "first" is the location of the lowest object
- * with an object name that could match "bin_pfx". See if we have
- * 0, 1 or more objects that actually match(es).
- */
- for (i = first; i < num; i++) {
- struct object_id oid;
-
- nth_packed_object_id(&oid, p, i);
- if (!match_hash(len, opts->prefix->hash, oid.hash))
- break;
-
- if (data->request) {
- struct object_info oi = *data->request;
- off_t offset = nth_packed_object_offset(p, i);
-
- ret = packed_object_info(source, p, offset, &oi);
- if (ret)
- goto out;
-
- ret = data->cb(&oid, &oi, data->cb_data);
- if (ret)
- goto out;
- } else {
- ret = data->cb(&oid, NULL, data->cb_data);
- if (ret)
- goto out;
+ for (uint32_t algo = 1; algo < GIT_HASH_NALGOS; algo++) {
+ const struct git_hash_algo *algop = &hash_algos[algo];
+
+ /*
+ * Skip this algorithm unless it's the desired one or we've
+ * asked for any algorithm.
+ */
+ if (opts->prefix->algo != GIT_HASH_UNKNOWN &&
+ opts->prefix->algo != algo)
+ continue;
+ /*
+ * Skip this algorithm unless it's the main algorithm or we have
+ * a compatibility algorithm and it's that algorithm.
+ */
+ if (algop != p->repo->hash_algo &&
+ (!p->repo->compat_hash_algo ||
+ algop != p->repo->compat_hash_algo))
+ continue;
+ if (p->index_version < 3 && algop != p->repo->hash_algo)
+ continue;
+
+ oidcpy(&wanted, opts->prefix);
+ wanted.algo = algo;
+
+ bsearch_pack(&wanted, p, &first);
+
+ /*
+ * At this point, "first" is the location of the lowest object
+ * with an object name that could match "bin_pfx". See if we have
+ * 0, 1 or more objects that actually match(es).
+ */
+ for (i = first; i < num; i++) {
+ struct object_id oid;
+
+ if (nth_packed_object_id_algop(&oid, p, i, algop, algop) ||
+ !match_hash(len, opts->prefix->hash, oid.hash))
+ break;
+
+ if (data->request) {
+ struct object_info oi = *data->request;
+ off_t offset = nth_packed_object_offset_algop(p, i, algop);
+
+ ret = packed_object_info(source, p, offset, &oi);
+ if (ret)
+ goto out;
+
+ ret = data->cb(&oid, &oi, data->cb_data);
+ if (ret)
+ goto out;
+ } else {
+ ret = data->cb(&oid, NULL, data->cb_data);
+ if (ret)
+ goto out;
+ }
}
}
diff --git a/packfile.c b/packfile.c
index 3bfd427c6c..3f3eb27e42 100644
--- a/packfile.c
+++ b/packfile.c
@@ -1919,16 +1919,27 @@ void check_pack_index_ptr(const struct packed_git *p, const void *vptr)
}
off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)
+{
+ return nth_packed_object_offset_algop(p, n, p->repo->hash_algo);
+}
+
+off_t nth_packed_object_offset_algop(const struct packed_git *p, uint32_t n,
+ const struct git_hash_algo *algop)
{
const unsigned char *index = p->index_data;
const unsigned int hashsz = p->repo->hash_algo->rawsz;
+
+ if (p->index_version < 3 && algop != p->repo->hash_algo)
+ BUG("pack index v%d only stores main object IDs",
+ p->index_version);
+
if (p->index_version == 1) {
index += 4 * 256;
return ntohl(*((uint32_t *)(index + st_mult(hashsz + 4, n))));
} else {
uint32_t off;
if (p->index_version == 3)
- n = nth_packed_object_pack_order_algop(p, n, p->repo->hash_algo);
+ n = nth_packed_object_pack_order_algop(p, n, algop);
index += st_add(p->crc_offset, st_mult(p->num_objects, 4));
off = ntohl(*((uint32_t *)(index + st_mult(4, n))));
if (!(off & 0x80000000))
diff --git a/packfile.h b/packfile.h
index 73452b2a43..40760295ee 100644
--- a/packfile.h
+++ b/packfile.h
@@ -303,6 +303,13 @@ int nth_packed_object_id_algop(struct object_id *, struct packed_git *,
*/
off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);
+/*
+ * The same as nth_packed_object_offset, but the nth object is counted in
+ * algop. For indices older than v3, algop must be the main algorithm.
+ */
+off_t nth_packed_object_offset_algop(const struct packed_git *, uint32_t n,
+ const struct git_hash_algo *algop);
+
/*
* Return the index into pack order of the nth object within the specified
* packfile. The index must already be opened and it must be a v3 index.
diff --git a/t/t1016-compatObjectFormat.sh b/t/t1016-compatObjectFormat.sh
index 030d6000b5..97a299f088 100755
--- a/t/t1016-compatObjectFormat.sh
+++ b/t/t1016-compatObjectFormat.sh
@@ -269,6 +269,36 @@ do
git -C map-repo rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
test_cmp expect actual
'
+
+ test_expect_success 'abbreviated compat object IDs are found using pack index v3' '
+ test_when_finished "rm -rf abbrev-repo v3.idx" &&
+ create_compat_repo $hash abbrev-repo &&
+ echo content >abbrev-repo/file &&
+ git -C abbrev-repo add file &&
+ git -C abbrev-repo commit --message A &&
+ compat_oid=$(git -C abbrev-repo rev-parse --output-object-format=$(compat_hash $hash) HEAD) &&
+ abbrev=$(echo $compat_oid | cut -c 1-12) &&
+
+ # A multi-pack-index only stores object IDs in the main
+ # algorithm.
+ GIT_TEST_MULTI_PACK_INDEX=0 git -C abbrev-repo repack -a -d &&
+ pack=$(echo abbrev-repo/.git/objects/pack/pack-*.pack) &&
+ git -C abbrev-repo index-pack --index-version=3 \
+ -o "$(pwd)/v3.idx" "$(pwd)/$pack" &&
+ rm abbrev-repo/.git/objects/loose-object-idx &&
+
+ # A v2 index only has object IDs in the main algorithm.
+ test_must_fail git -C abbrev-repo cat-file -t $abbrev &&
+
+ rm -f "${pack%.pack}.idx" "${pack%.pack}.rev" &&
+ mv v3.idx "${pack%.pack}.idx" &&
+ echo commit >expect &&
+ git -C abbrev-repo cat-file -t $abbrev >actual &&
+ test_cmp expect actual &&
+ echo $compat_oid >expect &&
+ git -C abbrev-repo rev-parse --output-object-format=$(compat_hash $hash) $abbrev >actual &&
+ test_cmp expect actual
+ '
done
cd "$base"
--
2.56.0.215.g47fdf2208c
^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v2 13/13] doc: move pack index v3 documentation
2026-10-23 7:25 [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Christian Couder
` (11 preceding siblings ...)
2026-10-23 7:25 ` [PATCH v2 12/13] packfile: allow looking up compat object IDs in packs Christian Couder
@ 2026-10-23 7:25 ` Christian Couder
2026-10-23 12:43 ` [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Ben Knoble
2026-10-23 15:00 ` Junio C Hamano
14 siblings, 0 replies; 17+ messages in thread
From: Christian Couder @ 2026-10-23 7:25 UTC (permalink / raw)
To: git
Cc: Junio C Hamano, brian m . carlson, Patrick Steinhardt,
Elijah Newren, Jeff King, Johannes Schindelin, Christian Couder
From: "brian m. carlson" <sandals@crustytoothpaste.net>
Right now, we have our pack index v3 documentation in the hash function
transition document. However, now that we've implemented pack index v3,
people will look for the index documentation in the normal place, so
copy it there.
We preserve it in the hash function transition document for
compatibility.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Christian Couder <christian.couder@gmail.com>
---
Documentation/gitformat-pack.adoc | 75 +++++++++++++++++++++++++++++++
1 file changed, 75 insertions(+)
diff --git a/Documentation/gitformat-pack.adoc b/Documentation/gitformat-pack.adoc
index 3416edceab..19afbc881e 100644
--- a/Documentation/gitformat-pack.adoc
+++ b/Documentation/gitformat-pack.adoc
@@ -318,6 +318,81 @@ Pack file entry: <+
Index checksum of all of the above.
+== Version 3 pack-*.idx files have the format:
+
+ - A header appears at the beginning and consists of the following:
+ * The 4-byte pack index signature: '\377t0c'
+ * 4-byte version number: 3
+ * 4-byte length of the header section, including the signature and
+ version number
+ * 4-byte number of objects contained in the pack
+ * 4-byte number of object formats in this pack index: 1 or 2
+ * For each object format:
+ ** 4-byte format identifier (e.g., 'sha1' for SHA-1)
+ ** 4-byte length in bytes of shortened object names. This is the
+ shortest possible length needed to make names in the shortened
+ object name table unambiguous.
+ ** 8-byte integer, recording where tables relating to this format
+ are stored in this index file, as an offset from the beginning.
+ * 8-byte offset to the trailer from the beginning of this file.
+ * Zero or more additional key/value pairs (4-byte key, 4-byte
+ value). Only one key is supported: 'PSRC'. See the "Loose objects
+ and unreachable objects" section of the hash function transition
+ document (`Documentation/technical/hash-function-transition.adoc`)
+ for supported values and how this is used. All other keys are reserved. Readers must ignore
+ unrecognized keys.
+ - Zero or more NUL bytes. This can optionally be used to improve the
+ alignment of the full object name table below.
+ - Tables for the first object format:
+ * A sorted table of shortened object names. These are prefixes of
+ the names of all objects in this pack file, packed together
+ without offset values to reduce the cache footprint of the binary
+ search for a specific object name.
+
+ * A table of full object names in pack order. This allows resolving
+ a reference to "the nth object in the pack file" (from a
+ reachability bitmap or from the next table of another object
+ format) to its object name.
+
+ * A table of 4-byte values mapping object name order to pack order.
+ For an object in the table of sorted shortened object names, the
+ value at the corresponding index in this table is the index in the
+ previous table for that same object.
+ This can be used to look up the object in reachability bitmaps or
+ to look up its name in another object format.
+
+ * A table of 4-byte CRC32 values of the packed object data, in the
+ order that the objects appear in the pack file. This is to allow
+ compressed data to be copied directly from pack to pack during
+ repacking without undetected data corruption.
+
+ * A table of 4-byte offset values. The index of this table in pack order
+ indicates where that object can be found in the pack file. These are
+ usually 31-bit pack file offsets, but large offsets are encoded as
+ an index into the next table with the most significant bit set.
+
+ * A table of 8-byte offset entries (empty for pack files less than
+ 2 GiB). Pack files are organized with heavily used objects toward
+ the front, so most object references should not need to refer to
+ this table.
+ - Zero or more NUL bytes.
+ - Tables for the second object format, with the same layout as above,
+ up to and not including the table of CRC32 values.
+ - Zero or more NUL bytes.
+ - The trailer consists of the following:
+ * A copy of the hash checksum at the end of the corresponding
+ packfile.
+
+ * Hash checksum of all of the above.
+
+Note that the two hashes at the end of the index file are in whatever the
+repository's main algorithm is. In the usual case when there are multiple
+algorithms, the main algorithm will be SHA-256 and the compatibility algorithm
+will be SHA-1. Git always writes the main algorithm first, but doesn't require this.
+
+All 4-byte numbers are in network order and must be 4-byte aligned in the file,
+so the NUL padding may be required in some cases.
+
== pack-*.rev files have the format:
- A 4-byte magic number '0x52494458' ('RIDX').
--
2.56.0.215.g47fdf2208c
^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3
2026-10-23 7:25 [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Christian Couder
` (12 preceding siblings ...)
2026-10-23 7:25 ` [PATCH v2 13/13] doc: move pack index v3 documentation Christian Couder
@ 2026-10-23 12:43 ` Ben Knoble
2026-10-23 15:00 ` Junio C Hamano
14 siblings, 0 replies; 17+ messages in thread
From: Ben Knoble @ 2026-10-23 12:43 UTC (permalink / raw)
To: Christian Couder
Cc: git, Junio C Hamano, brian m carlson, Patrick Steinhardt,
Elijah Newren, Jeff King, Johannes Schindelin, Christian Couder
> Le 23 oct. 2026 à 03:30, Christian Couder <christian.couder@gmail.com> a écrit :
>
> This series is the next part of brian m. carlson's SHA-1/SHA-256
> interoperability work.
>
> ## Changes compared to v1
>
> Thanks to Junio for his comments on this series.
>
> The only change compared to v1 is that I added my `Signed-off-by: ...`
> to patches 2, 3 and 5 (that are otherwise unchanged compared to
> brian's original patches).
In such a long cover letter, it might be helpful to present the changes from prior versions much earlier ;)
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3
2026-10-23 7:25 [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Christian Couder
` (13 preceding siblings ...)
2026-10-23 12:43 ` [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Ben Knoble
@ 2026-10-23 15:00 ` Junio C Hamano
2026-10-27 12:53 ` Christian Couder
14 siblings, 1 reply; 17+ messages in thread
From: Junio C Hamano @ 2026-10-23 15:00 UTC (permalink / raw)
To: Christian Couder
Cc: git, brian m . carlson, Patrick Steinhardt, Elijah Newren,
Jeff King, Johannes Schindelin
Christian Couder <christian.couder@gmail.com> writes:
> This series is the next part of brian m. carlson's SHA-1/SHA-256
> interoperability work.
>
> It adds support for reading and writing pack index v3, which stores
> object IDs in both the main and the compatibility hash
> algorithms. Interoperability needs it so that packed objects can be
> mapped between algorithms, the same way the loose object map already
> does for loose objects.
> ...
> My `Signed-off-by: ...` was added to all the patches. AI assistance
> (mostly Claude Opus 5.5) was used.
> ...
> Except that I added my `Signed-off-by: ...` to them, patches 2, 3 and
> 5 are unchanged.
> ## Changes compared to v1
>
> Thanks to Junio for his comments on this series.
>
> The only change compared to v1 is that I added my `Signed-off-by: ...`
> to patches 2, 3 and 5 (that are otherwise unchanged compared to
> brian's original patches).
Will replace.
Next time please make sure that the cover letter for a new
round is a reply to the cover letter of either immediate previous
iteration or the first iteration (and all patches replies to the
cover letter of the new round). When you send a new round as a
new thread unrelated that is unrelated to the previous round's
patches, automation will not quite work well.
Thanks.
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3
2026-10-23 15:00 ` Junio C Hamano
@ 2026-10-27 12:53 ` Christian Couder
0 siblings, 0 replies; 17+ messages in thread
From: Christian Couder @ 2026-10-27 12:53 UTC (permalink / raw)
To: Junio C Hamano
Cc: git, brian m . carlson, Patrick Steinhardt, Elijah Newren,
Jeff King, Johannes Schindelin
On Fri, Oct 23, 2026 at 5:00 PM Junio C Hamano <gitster@pobox.com> wrote:
> Will replace.
>
> Next time please make sure that the cover letter for a new
> round is a reply to the cover letter of either immediate previous
> iteration or the first iteration (and all patches replies to the
> cover letter of the new round). When you send a new round as a
> new thread unrelated that is unrelated to the previous round's
> patches, automation will not quite work well.
Yeah, sorry. I am going to try a new approach using a
sendemail-validate hook and scripts (see
https://gitlab.com/chriscool/check-send-email), which can also be used
with b4, to avoid most common mistakes.
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2026-10-27 12:53 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-23 7:25 [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Christian Couder
2026-10-23 7:25 ` [PATCH v2 01/13] loose: allow looking up in object map with source Christian Couder
2026-10-23 7:25 ` [PATCH v2 02/13] loose: rename repo_add_loose_object_map Christian Couder
2026-10-23 7:25 ` [PATCH v2 03/13] loose: add a flag to control whether we write mapped OIDs Christian Couder
2026-10-23 7:25 ` [PATCH v2 04/13] odb: implement interoperability functions Christian Couder
2026-10-23 7:25 ` [PATCH v2 05/13] object-file-convert: implement mapping in terms of object DB Christian Couder
2026-10-23 7:25 ` [PATCH v2 06/13] pack-write: add support for pack index v3 Christian Couder
2026-10-23 7:25 ` [PATCH v2 07/13] builtin: populate pack index v3 fields when indexing Christian Couder
2026-10-23 7:25 ` [PATCH v2 08/13] object-file: handle compatibility algorithm Christian Couder
2026-10-23 7:25 ` [PATCH v2 09/13] packfile: add support for reading pack index v3 Christian Couder
2026-10-23 7:25 ` [PATCH v2 10/13] packfile: validate pack index v3 tables Christian Couder
2026-10-23 7:25 ` [PATCH v2 11/13] loose: map object IDs using pack index v3 Christian Couder
2026-10-23 7:25 ` [PATCH v2 12/13] packfile: allow looking up compat object IDs in packs Christian Couder
2026-10-23 7:25 ` [PATCH v2 13/13] doc: move pack index v3 documentation Christian Couder
2026-10-23 12:43 ` [PATCH v2 00/13] SHA-1/SHA-256 interoperability, part 3: pack index v3 Ben Knoble
2026-10-23 15:00 ` Junio C Hamano
2026-10-27 12:53 ` Christian Couder
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox