* [PATCH 16/30] object-file-convert: convert tag objects when writing
From: Eric W. Biederman @ 2023-09-27 19:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson, Eric W . Biederman
In-Reply-To: <87jzsbjt0a.fsf@gmail.froward.int.ebiederm.org>
From: "brian m. carlson" <sandals@crustytoothpaste.net>
When writing a tag object in a repository with both SHA-1 and SHA-256,
we'll need to convert our commit objects so that we can write the hash
values for both into the repository. To do so, let's add a function to
convert tag objects.
Note that signatures for tag objects in the current algorithm trail the
message, and those for the alternate algorithm are in headers.
Therefore, we parse the tag object for both a trailing signature and a
header and then, when writing the other format, swap the two around.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
---
object-file-convert.c | 52 ++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 51 insertions(+), 1 deletion(-)
diff --git a/object-file-convert.c b/object-file-convert.c
index a9e7a208a707..777ae5b58036 100644
--- a/object-file-convert.c
+++ b/object-file-convert.c
@@ -7,6 +7,8 @@
#include "hash.h"
#include "object.h"
#include "loose.h"
+#include "commit.h"
+#include "gpg-interface.h"
#include "object-file-convert.h"
int repo_oid_to_algop(struct repository *repo, const struct object_id *src,
@@ -83,6 +85,52 @@ static int convert_tree_object(struct strbuf *out,
return 0;
}
+static int convert_tag_object(struct strbuf *out,
+ const struct git_hash_algo *from,
+ const struct git_hash_algo *to,
+ const char *buffer, size_t size)
+{
+ struct strbuf payload = STRBUF_INIT, temp = STRBUF_INIT, oursig = STRBUF_INIT, othersig = STRBUF_INIT;
+ size_t payload_size;
+ struct object_id oid, mapped_oid;
+ const char *p;
+
+ /* Add some slop for longer signature header in the new algorithm. */
+ strbuf_grow(out, size + 7);
+
+ /* Is there a signature for our algorithm? */
+ payload_size = parse_signed_buffer(buffer, size);
+ strbuf_add(&payload, buffer, payload_size);
+ if (payload_size != size) {
+ /* Yes, there is. */
+ strbuf_add(&oursig, buffer + payload_size, size - payload_size);
+ }
+ /* Now, is there a signature for the other algorithm? */
+ if (parse_buffer_signed_by_header(payload.buf, payload.len, &temp, &othersig, to)) {
+ /* Yes, there is. */
+ strbuf_swap(&payload, &temp);
+ strbuf_release(&temp);
+ }
+
+ /*
+ * Our payload is now in payload and we may have up to two signatrures
+ * in oursig and othersig.
+ */
+ if (strncmp(payload.buf, "object ", 7) || payload.buf[from->hexsz + 7] != '\n')
+ return error("bogus tag object");
+ if (parse_oid_hex_algop(payload.buf + 7, &oid, &p, from) < 0)
+ return error("bad tag object ID");
+ if (repo_oid_to_algop(the_repository, &oid, to, &mapped_oid))
+ return error("unable to map tree %s in tag object",
+ oid_to_hex(&oid));
+ strbuf_addf(out, "object %s", oid_to_hex(&mapped_oid));
+ strbuf_add(out, p, payload.len - (p - payload.buf));
+ strbuf_addbuf(out, &othersig);
+ if (oursig.len)
+ add_header_signature(out, &oursig, from);
+ return 0;
+}
+
int convert_object_file(struct strbuf *outbuf,
const struct git_hash_algo *from,
const struct git_hash_algo *to,
@@ -100,8 +148,10 @@ int convert_object_file(struct strbuf *outbuf,
case OBJ_TREE:
ret = convert_tree_object(outbuf, from, to, buf, len);
break;
- case OBJ_COMMIT:
case OBJ_TAG:
+ ret = convert_tag_object(outbuf, from, to, buf, len);
+ break;
+ case OBJ_COMMIT:
default:
/* Not implemented yet, so fail. */
ret = -1;
--
2.41.0
^ permalink raw reply related
* [PATCH 15/30] object-file-convert: add a function to convert trees between algorithms
From: Eric W. Biederman @ 2023-09-27 19:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson, Eric W . Biederman
In-Reply-To: <87jzsbjt0a.fsf@gmail.froward.int.ebiederm.org>
From: "brian m. carlson" <sandals@crustytoothpaste.net>
In the future, we're going to want to provide SHA-256 repositories that
have compatibility support for SHA-1 as well. In order to do so, we'll
need to be able to convert tree objects from SHA-256 to SHA-1 by writing
a tree with each SHA-256 object ID mapped to a SHA-1 object ID.
We implement a function, convert_tree_object, that takes an existing
tree buffer and writes it to a new strbuf, converting between
algorithms. Let's make this function generic, because while we only
need it to convert from the main algorithm to the compatibility
algorithm now, we may need to do the other way around in the future,
such as for transport.
We avoid reusing the code in decode_tree_entry because that code
normalizes data, and we don't want that here. We want to produce a
complete round trip of data, so if, for example, the old entry had a
wrongly zero-padded mode, we'd want to preserve that when converting to
ensure a stable hash value.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
---
object-file-convert.c | 51 ++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 50 insertions(+), 1 deletion(-)
diff --git a/object-file-convert.c b/object-file-convert.c
index 4d62ed192bf0..a9e7a208a707 100644
--- a/object-file-convert.c
+++ b/object-file-convert.c
@@ -1,8 +1,10 @@
#include "git-compat-util.h"
#include "gettext.h"
#include "strbuf.h"
+#include "hex.h"
#include "repository.h"
#include "hash-ll.h"
+#include "hash.h"
#include "object.h"
#include "loose.h"
#include "object-file-convert.h"
@@ -36,6 +38,51 @@ int repo_oid_to_algop(struct repository *repo, const struct object_id *src,
return 0;
}
+static int decode_tree_entry_raw(struct object_id *oid, const char **path,
+ size_t *len, const struct git_hash_algo *algo,
+ const char *buf, unsigned long size)
+{
+ uint16_t mode;
+ const unsigned hashsz = algo->rawsz;
+
+ if (size < hashsz + 3 || buf[size - (hashsz + 1)]) {
+ return -1;
+ }
+
+ *path = parse_mode(buf, &mode);
+ if (!*path || !**path)
+ return -1;
+ *len = strlen(*path) + 1;
+
+ oidread_algop(oid, (const unsigned char *)*path + *len, algo);
+ return 0;
+}
+
+static int convert_tree_object(struct strbuf *out,
+ const struct git_hash_algo *from,
+ const struct git_hash_algo *to,
+ const char *buffer, size_t size)
+{
+ const char *p = buffer, *end = buffer + size;
+
+ while (p < end) {
+ struct object_id entry_oid, mapped_oid;
+ const char *path = NULL;
+ size_t pathlen;
+
+ if (decode_tree_entry_raw(&entry_oid, &path, &pathlen, from, p,
+ end - p))
+ return error(_("failed to decode tree entry"));
+ if (repo_oid_to_algop(the_repository, &entry_oid, to, &mapped_oid))
+ return error(_("failed to map tree entry for %s"), oid_to_hex(&entry_oid));
+ strbuf_add(out, p, path - p);
+ strbuf_add(out, path, pathlen);
+ strbuf_add(out, mapped_oid.hash, to->rawsz);
+ p = path + pathlen + from->rawsz;
+ }
+ return 0;
+}
+
int convert_object_file(struct strbuf *outbuf,
const struct git_hash_algo *from,
const struct git_hash_algo *to,
@@ -50,8 +97,10 @@ int convert_object_file(struct strbuf *outbuf,
die("Refusing noop object file conversion");
switch (type) {
- case OBJ_COMMIT:
case OBJ_TREE:
+ ret = convert_tree_object(outbuf, from, to, buf, len);
+ break;
+ case OBJ_COMMIT:
case OBJ_TAG:
default:
/* Not implemented yet, so fail. */
--
2.41.0
^ permalink raw reply related
* [PATCH 13/30] cache: add a function to read an OID of a specific algorithm
From: Eric W. Biederman @ 2023-09-27 19:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson, Eric W. Biederman
In-Reply-To: <87jzsbjt0a.fsf@gmail.froward.int.ebiederm.org>
From: "brian m. carlson" <sandals@crustytoothpaste.net>
Currently, we always read a object ID of the current algorithm with
oidread. However, once we start converting objects, we'll need to
consider what happens when we want to read an object ID of a specific
algorithm, such as the compatibility algorithm. To make this easier,
let's define oidread_algop, which specifies which algorithm we should
use for our object ID, and define oidread in terms of it.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
---
hash.h | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/hash.h b/hash.h
index 615ae0691d07..e064807c1733 100644
--- a/hash.h
+++ b/hash.h
@@ -73,10 +73,15 @@ static inline void oidclr(struct object_id *oid)
oid->algo = hash_algo_by_ptr(the_hash_algo);
}
+static inline void oidread_algop(struct object_id *oid, const unsigned char *hash, const struct git_hash_algo *algop)
+{
+ memcpy(oid->hash, hash, algop->rawsz);
+ oid->algo = hash_algo_by_ptr(algop);
+}
+
static inline void oidread(struct object_id *oid, const unsigned char *hash)
{
- memcpy(oid->hash, hash, the_hash_algo->rawsz);
- oid->algo = hash_algo_by_ptr(the_hash_algo);
+ oidread_algop(oid, hash, the_hash_algo);
}
static inline int is_empty_blob_sha1(const unsigned char *sha1)
--
2.41.0
^ permalink raw reply related
* [PATCH 14/30] object: Factor out parse_mode out of fast-import and tree-walk into in object.h
From: Eric W. Biederman @ 2023-09-27 19:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson, Eric W. Biederman
In-Reply-To: <87jzsbjt0a.fsf@gmail.froward.int.ebiederm.org>
From: "Eric W. Biederman" <ebiederm@xmission.com>
builtin/fast-import.c and tree-walk.c have almost identical version of
get_mode. The two functions started out the same but have diverged
slightly. The version in fast-import changed mode to a uint16_t to
save memory. The version in tree-walk started erroring if no mode was
present.
As far as I can tell both of these changes are valid for both of the
callers, so add the both changes and place the common parsing helper
in object.h
Rename the helper from get_mode to parse_mode so it does not
conflict with another helper named get_mode in diff-no-index.c
This will be used shortly in a new helper decode_tree_entry_raw
which is used to compute cmpatibility objects as part of
the sha256 transition.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
---
builtin/fast-import.c | 18 ++----------------
object.h | 18 ++++++++++++++++++
tree-walk.c | 22 +++-------------------
3 files changed, 23 insertions(+), 35 deletions(-)
diff --git a/builtin/fast-import.c b/builtin/fast-import.c
index 4dbb10aff3da..2c645fcfbe3f 100644
--- a/builtin/fast-import.c
+++ b/builtin/fast-import.c
@@ -1235,20 +1235,6 @@ static void *gfi_unpack_entry(
return unpack_entry(the_repository, p, oe->idx.offset, &type, sizep);
}
-static const char *get_mode(const char *str, uint16_t *modep)
-{
- unsigned char c;
- uint16_t mode = 0;
-
- while ((c = *str++) != ' ') {
- if (c < '0' || c > '7')
- return NULL;
- mode = (mode << 3) + (c - '0');
- }
- *modep = mode;
- return str;
-}
-
static void load_tree(struct tree_entry *root)
{
struct object_id *oid = &root->versions[1].oid;
@@ -1286,7 +1272,7 @@ static void load_tree(struct tree_entry *root)
t->entries[t->entry_count++] = e;
e->tree = NULL;
- c = get_mode(c, &e->versions[1].mode);
+ c = parse_mode(c, &e->versions[1].mode);
if (!c)
die("Corrupt mode in %s", oid_to_hex(oid));
e->versions[0].mode = e->versions[1].mode;
@@ -2275,7 +2261,7 @@ static void file_change_m(const char *p, struct branch *b)
struct object_id oid;
uint16_t mode, inline_data = 0;
- p = get_mode(p, &mode);
+ p = parse_mode(p, &mode);
if (!p)
die("Corrupt mode: %s", command_buf.buf);
switch (mode) {
diff --git a/object.h b/object.h
index 114d45954d08..70c8d4ae63dc 100644
--- a/object.h
+++ b/object.h
@@ -190,6 +190,24 @@ void *create_object(struct repository *r, const struct object_id *oid, void *obj
void *object_as_type(struct object *obj, enum object_type type, int quiet);
+
+static inline const char *parse_mode(const char *str, uint16_t *modep)
+{
+ unsigned char c;
+ unsigned int mode = 0;
+
+ if (*str == ' ')
+ return NULL;
+
+ while ((c = *str++) != ' ') {
+ if (c < '0' || c > '7')
+ return NULL;
+ mode = (mode << 3) + (c - '0');
+ }
+ *modep = mode;
+ return str;
+}
+
/*
* Returns the object, having parsed it to find out what it is.
*
diff --git a/tree-walk.c b/tree-walk.c
index 29ead71be173..3af50a01c2c7 100644
--- a/tree-walk.c
+++ b/tree-walk.c
@@ -10,27 +10,11 @@
#include "pathspec.h"
#include "json-writer.h"
-static const char *get_mode(const char *str, unsigned int *modep)
-{
- unsigned char c;
- unsigned int mode = 0;
-
- if (*str == ' ')
- return NULL;
-
- while ((c = *str++) != ' ') {
- if (c < '0' || c > '7')
- return NULL;
- mode = (mode << 3) + (c - '0');
- }
- *modep = mode;
- return str;
-}
-
static int decode_tree_entry(struct tree_desc *desc, const char *buf, unsigned long size, struct strbuf *err)
{
const char *path;
- unsigned int mode, len;
+ unsigned int len;
+ uint16_t mode;
const unsigned hashsz = the_hash_algo->rawsz;
if (size < hashsz + 3 || buf[size - (hashsz + 1)]) {
@@ -38,7 +22,7 @@ static int decode_tree_entry(struct tree_desc *desc, const char *buf, unsigned l
return -1;
}
- path = get_mode(buf, &mode);
+ path = parse_mode(buf, &mode);
if (!path) {
strbuf_addstr(err, _("malformed mode in tree entry"));
return -1;
--
2.41.0
^ permalink raw reply related
* [PATCH 12/30] tag: sign both hashes
From: Eric W. Biederman @ 2023-09-27 19:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson, Eric W. Biederman
In-Reply-To: <87jzsbjt0a.fsf@gmail.froward.int.ebiederm.org>
From: "Eric W. Biederman" <ebiederm@xmission.com>
When we write a tag the object oid is specific to the hash algorithm.
This matters when a tag is signed. The hash transition plan calls for
signatures on both the sha1 form and the sha256 form of the object,
and for both of those signatures to live in the tag object.
To generate tag object with multiple signatures, first compute the
unsigned form of the tag, and then if the tag is being signed compute
the unsigned form of the tag with the compatibilityr hash. Then
compute compute the signatures of both buffers.
Once the signatures are computed add them to both buffers. This
allows computing the compatibility hash in do_sign, saving
write_object_file the expense of recomputing the compatibility tag
just to compute it's hash.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
---
builtin/tag.c | 45 +++++++++++++++++++++++++++++++++++++++++----
1 file changed, 41 insertions(+), 4 deletions(-)
diff --git a/builtin/tag.c b/builtin/tag.c
index 3918eacbb57b..8c4bc28952c2 100644
--- a/builtin/tag.c
+++ b/builtin/tag.c
@@ -28,6 +28,7 @@
#include "ref-filter.h"
#include "date.h"
#include "write-or-die.h"
+#include "object-file-convert.h"
static const char * const git_tag_usage[] = {
N_("git tag [-a | -s | -u <key-id>] [-f] [-m <msg> | -F <file>] [-e]\n"
@@ -174,9 +175,43 @@ static int verify_tag(const char *name, const char *ref UNUSED,
return 0;
}
-static int do_sign(struct strbuf *buffer)
+static int do_sign(struct strbuf *buffer, struct object_id **compat_oid,
+ struct object_id *compat_oid_buf)
{
- return sign_buffer(buffer, buffer, get_signing_key());
+ const struct git_hash_algo *compat = the_repository->compat_hash_algo;
+ struct strbuf sig = STRBUF_INIT, compat_sig = STRBUF_INIT;
+ struct strbuf compat_buf = STRBUF_INIT;
+ const char *keyid = get_signing_key();
+ int ret = -1;
+
+ if (sign_buffer(buffer, &sig, keyid))
+ return -1;
+
+ if (compat) {
+ const struct git_hash_algo *algo = the_repository->hash_algo;
+
+ if (convert_object_file(&compat_buf, algo, compat,
+ buffer->buf, buffer->len, OBJ_TAG, 1))
+ goto out;
+ if (sign_buffer(&compat_buf, &compat_sig, keyid))
+ goto out;
+ add_header_signature(&compat_buf, &sig, algo);
+ strbuf_addbuf(&compat_buf, &compat_sig);
+ hash_object_file(compat, compat_buf.buf, compat_buf.len,
+ OBJ_TAG, compat_oid_buf);
+ *compat_oid = compat_oid_buf;
+ }
+
+ if (compat_sig.len)
+ add_header_signature(buffer, &compat_sig, compat);
+
+ strbuf_addbuf(buffer, &sig);
+ ret = 0;
+out:
+ strbuf_release(&sig);
+ strbuf_release(&compat_sig);
+ strbuf_release(&compat_buf);
+ return ret;
}
static const char tag_template[] =
@@ -249,9 +284,11 @@ static void write_tag_body(int fd, const struct object_id *oid)
static int build_tag_object(struct strbuf *buf, int sign, struct object_id *result)
{
- if (sign && do_sign(buf) < 0)
+ struct object_id *compat_oid = NULL, compat_oid_buf;
+ if (sign && do_sign(buf, &compat_oid, &compat_oid_buf) < 0)
return error(_("unable to sign the tag"));
- if (write_object_file(buf->buf, buf->len, OBJ_TAG, result) < 0)
+ if (write_object_file_flags(buf->buf, buf->len, OBJ_TAG, result,
+ compat_oid, 0) < 0)
return error(_("unable to write tag file"));
return 0;
}
--
2.41.0
^ permalink raw reply related
* [PATCH 11/30] commit: Export add_header_signature to support handling signatures on tags
From: Eric W. Biederman @ 2023-09-27 19:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson, Eric W. Biederman
In-Reply-To: <87jzsbjt0a.fsf@gmail.froward.int.ebiederm.org>
From: "Eric W. Biederman" <ebiederm@xmission.com>
Rename add_commit_signature as add_header_signature, and expose it so
that it can be used for converting tags from one object format to
another.
Inspired-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
---
commit.c | 6 +++---
commit.h | 1 +
2 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/commit.c b/commit.c
index a6dac9a1957b..e75270171bc3 100644
--- a/commit.c
+++ b/commit.c
@@ -1101,7 +1101,7 @@ static const char *gpg_sig_headers[] = {
"gpgsig-sha256",
};
-static int add_commit_signature(struct strbuf *buf, struct strbuf *sig, const struct git_hash_algo *algo)
+int add_header_signature(struct strbuf *buf, struct strbuf *sig, const struct git_hash_algo *algo)
{
int inspos, copypos;
const char *eoh;
@@ -1769,9 +1769,9 @@ int commit_tree_extended(const char *msg, size_t msg_len,
for (i = 0; i < ARRAY_SIZE(bufs); i++) {
if (!bufs[i].algo)
continue;
- add_commit_signature(&buffer, bufs[i].sig, bufs[i].algo);
+ add_header_signature(&buffer, bufs[i].sig, bufs[i].algo);
if (r->compat_hash_algo)
- add_commit_signature(&compat_buffer, bufs[i].sig, bufs[i].algo);
+ add_header_signature(&compat_buffer, bufs[i].sig, bufs[i].algo);
}
}
diff --git a/commit.h b/commit.h
index 28928833c544..03edcec0129f 100644
--- a/commit.h
+++ b/commit.h
@@ -370,5 +370,6 @@ int parse_buffer_signed_by_header(const char *buffer,
struct strbuf *payload,
struct strbuf *signature,
const struct git_hash_algo *algop);
+int add_header_signature(struct strbuf *buf, struct strbuf *sig, const struct git_hash_algo *algo);
#endif /* COMMIT_H */
--
2.41.0
^ permalink raw reply related
* [PATCH 10/30] commit: Convert mergetag before computing the signature of a commit
From: Eric W. Biederman @ 2023-09-27 19:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson, Eric W. Biederman
In-Reply-To: <87jzsbjt0a.fsf@gmail.froward.int.ebiederm.org>
From: "Eric W. Biederman" <ebiederm@xmission.com>
It so happens that commit mergetag lines embed a tag object. So to
compute the compatible signature of a commit object that has mergetag
lines the compatible embedded tag must be computed first.
Implement this by duplicating and converting the commit extra headers
into the compatible version of the commit extra headers, that need
to be passed to commit_tree_extended.
To handle merge tags only the compatible extra headers need to be
computed.
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
---
commit.c | 42 +++++++++++++++++++++++++++++++++++++++++-
1 file changed, 41 insertions(+), 1 deletion(-)
diff --git a/commit.c b/commit.c
index 46696ede8981..a6dac9a1957b 100644
--- a/commit.c
+++ b/commit.c
@@ -1355,6 +1355,39 @@ void append_merge_tag_headers(struct commit_list *parents,
}
}
+static int convert_commit_extra_headers(struct commit_extra_header *orig,
+ struct commit_extra_header **result)
+{
+ const struct git_hash_algo *compat = the_repository->compat_hash_algo;
+ const struct git_hash_algo *algo = the_repository->hash_algo;
+ struct commit_extra_header *extra = NULL, **tail = &extra;
+ struct strbuf out = STRBUF_INIT;
+ while (orig) {
+ struct commit_extra_header *new;
+ CALLOC_ARRAY(new, 1);
+ if (!strcmp(orig->key, "mergetag")) {
+ if (convert_object_file(&out, algo, compat,
+ orig->value, orig->len,
+ OBJ_TAG, 1)) {
+ free(new);
+ free_commit_extra_headers(extra);
+ return -1;
+ }
+ new->key = xstrdup("mergetag");
+ new->value = strbuf_detach(&out, &new->len);
+ } else {
+ new->key = xstrdup(orig->key);
+ new->len = orig->len;
+ new->value = xmemdupz(orig->value, orig->len);
+ }
+ *tail = new;
+ tail = &new->next;
+ orig = orig->next;
+ }
+ *result = extra;
+ return 0;
+}
+
static void add_extra_header(struct strbuf *buffer,
struct commit_extra_header *extra)
{
@@ -1682,6 +1715,7 @@ int commit_tree_extended(const char *msg, size_t msg_len,
goto out;
}
if (r->compat_hash_algo) {
+ struct commit_extra_header *compat_extra = NULL;
struct object_id mapped_tree;
struct object_id *mapped_parents = xcalloc(nparents, sizeof(*mapped_parents));
if (repo_oid_to_algop(r, tree, r->compat_hash_algo, &mapped_tree)) {
@@ -1695,8 +1729,14 @@ int commit_tree_extended(const char *msg, size_t msg_len,
free(mapped_parents);
goto out;
}
+ if (convert_commit_extra_headers(extra, &compat_extra)) {
+ result = -1;
+ free(mapped_parents);
+ goto out;
+ }
write_commit_tree(&compat_buffer, msg, msg_len, &mapped_tree,
- mapped_parents, nparents, author, committer, extra);
+ mapped_parents, nparents, author, committer, compat_extra);
+ free_commit_extra_headers(compat_extra);
if (sign_commit && sign_commit_to_strbuf(&compat_sig, &compat_buffer, sign_commit)) {
result = -1;
--
2.41.0
^ permalink raw reply related
* [PATCH 09/30] commit: write commits for both hashes
From: Eric W. Biederman @ 2023-09-27 19:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson, Eric W . Biederman
In-Reply-To: <87jzsbjt0a.fsf@gmail.froward.int.ebiederm.org>
From: "brian m. carlson" <sandals@crustytoothpaste.net>
When we write a commit, we include data that is specific to the hash
algorithm, such as parents and the root tree. In order to write both a
SHA-1 commit and a SHA-256 version, we need to convert between them.
However, a straightforward conversion isn't necessarily what we want.
When we sign a commit, we sign its data, so if we create a commit for
SHA-256 and then write a SHA-1 version, we'll still have only signed the
SHA-256 data. While this is valid, it would be better to sign both
forms of data so people using SHA-1 can verify the signatures as well.
Consequently, we don't want to use the standard mapping that occurs when
we write an object. Instead, let's move most of the writing of the
commit into a separate function which is agnostic of the hash algorithm
and which simply writes into a buffer and specify both versions of the
object ourselves.
We can then call this function twice: once with the SHA-256 contents,
and if SHA-1 is enabled, once with the SHA-1 contents. If we're signing
the commit, we then sign both versions and append both signatures to
both buffers. To produce a consistent hash, we always append the
signatures in the order in which Git implemented them: first SHA-1, then
SHA-256.
In order to make this signing code work, we split the commit signing
code into two functions, one which signs the buffer, and one which
appends the signature.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
---
commit.c | 179 +++++++++++++++++++++++++++++++++++++++++--------------
1 file changed, 134 insertions(+), 45 deletions(-)
diff --git a/commit.c b/commit.c
index b3223478bc2a..46696ede8981 100644
--- a/commit.c
+++ b/commit.c
@@ -28,6 +28,7 @@
#include "shallow.h"
#include "tree.h"
#include "hook.h"
+#include "object-file-convert.h"
static struct commit_extra_header *read_commit_extra_header_lines(const char *buf, size_t len, const char **);
@@ -1100,12 +1101,11 @@ static const char *gpg_sig_headers[] = {
"gpgsig-sha256",
};
-int sign_with_header(struct strbuf *buf, const char *keyid)
+static int add_commit_signature(struct strbuf *buf, struct strbuf *sig, const struct git_hash_algo *algo)
{
- struct strbuf sig = STRBUF_INIT;
int inspos, copypos;
const char *eoh;
- const char *gpg_sig_header = gpg_sig_headers[hash_algo_by_ptr(the_hash_algo)];
+ const char *gpg_sig_header = gpg_sig_headers[hash_algo_by_ptr(algo)];
int gpg_sig_header_len = strlen(gpg_sig_header);
/* find the end of the header */
@@ -1115,15 +1115,8 @@ int sign_with_header(struct strbuf *buf, const char *keyid)
else
inspos = eoh - buf->buf + 1;
- if (!keyid || !*keyid)
- keyid = get_signing_key();
- if (sign_buffer(buf, &sig, keyid)) {
- strbuf_release(&sig);
- return -1;
- }
-
- for (copypos = 0; sig.buf[copypos]; ) {
- const char *bol = sig.buf + copypos;
+ for (copypos = 0; sig->buf[copypos]; ) {
+ const char *bol = sig->buf + copypos;
const char *eol = strchrnul(bol, '\n');
int len = (eol - bol) + !!*eol;
@@ -1136,11 +1129,17 @@ int sign_with_header(struct strbuf *buf, const char *keyid)
inspos += len;
copypos += len;
}
- strbuf_release(&sig);
return 0;
}
-
+static int sign_commit_to_strbuf(struct strbuf *sig, struct strbuf *buf, const char *keyid)
+{
+ if (!keyid || !*keyid)
+ keyid = get_signing_key();
+ if (sign_buffer(buf, sig, keyid))
+ return -1;
+ return 0;
+}
int parse_signed_commit(const struct commit *commit,
struct strbuf *payload, struct strbuf *signature,
@@ -1599,70 +1598,160 @@ N_("Warning: commit message did not conform to UTF-8.\n"
"You may want to amend it after fixing the message, or set the config\n"
"variable i18n.commitEncoding to the encoding your project uses.\n");
-int commit_tree_extended(const char *msg, size_t msg_len,
- const struct object_id *tree,
- struct commit_list *parents, struct object_id *ret,
- const char *author, const char *committer,
- const char *sign_commit,
- struct commit_extra_header *extra)
+static void write_commit_tree(struct strbuf *buffer, const char *msg, size_t msg_len,
+ const struct object_id *tree,
+ const struct object_id *parents, size_t parents_len,
+ const char *author, const char *committer,
+ struct commit_extra_header *extra)
{
- int result;
int encoding_is_utf8;
- struct strbuf buffer;
-
- assert_oid_type(tree, OBJ_TREE);
-
- if (memchr(msg, '\0', msg_len))
- return error("a NUL byte in commit log message not allowed.");
+ size_t i;
/* Not having i18n.commitencoding is the same as having utf-8 */
encoding_is_utf8 = is_encoding_utf8(git_commit_encoding);
- strbuf_init(&buffer, 8192); /* should avoid reallocs for the headers */
- strbuf_addf(&buffer, "tree %s\n", oid_to_hex(tree));
+ strbuf_init(buffer, 8192); /* should avoid reallocs for the headers */
+ strbuf_addf(buffer, "tree %s\n", oid_to_hex(tree));
/*
* NOTE! This ordering means that the same exact tree merged with a
* different order of parents will be a _different_ changeset even
* if everything else stays the same.
*/
- while (parents) {
- struct commit *parent = pop_commit(&parents);
- strbuf_addf(&buffer, "parent %s\n",
- oid_to_hex(&parent->object.oid));
- }
+ for (i = 0; i < parents_len; i++)
+ strbuf_addf(buffer, "parent %s\n", oid_to_hex(&parents[i]));
/* Person/date information */
if (!author)
author = git_author_info(IDENT_STRICT);
- strbuf_addf(&buffer, "author %s\n", author);
+ strbuf_addf(buffer, "author %s\n", author);
if (!committer)
committer = git_committer_info(IDENT_STRICT);
- strbuf_addf(&buffer, "committer %s\n", committer);
+ strbuf_addf(buffer, "committer %s\n", committer);
if (!encoding_is_utf8)
- strbuf_addf(&buffer, "encoding %s\n", git_commit_encoding);
+ strbuf_addf(buffer, "encoding %s\n", git_commit_encoding);
while (extra) {
- add_extra_header(&buffer, extra);
+ add_extra_header(buffer, extra);
extra = extra->next;
}
- strbuf_addch(&buffer, '\n');
+ strbuf_addch(buffer, '\n');
/* And add the comment */
- strbuf_add(&buffer, msg, msg_len);
+ strbuf_add(buffer, msg, msg_len);
+}
- /* And check the encoding */
- if (encoding_is_utf8 && !verify_utf8(&buffer))
- fprintf(stderr, _(commit_utf8_warn));
+int commit_tree_extended(const char *msg, size_t msg_len,
+ const struct object_id *tree,
+ struct commit_list *parents, struct object_id *ret,
+ const char *author, const char *committer,
+ const char *sign_commit,
+ struct commit_extra_header *extra)
+{
+ struct repository *r = the_repository;
+ int result = 0;
+ int encoding_is_utf8;
+ struct strbuf buffer, compat_buffer;
+ struct strbuf sig = STRBUF_INIT, compat_sig = STRBUF_INIT;
+ struct object_id *parent_buf = NULL, *compat_oid = NULL;
+ struct object_id compat_oid_buf;
+ size_t i, nparents;
+
+ /* Not having i18n.commitencoding is the same as having utf-8 */
+ encoding_is_utf8 = is_encoding_utf8(git_commit_encoding);
+
+ assert_oid_type(tree, OBJ_TREE);
+
+ if (memchr(msg, '\0', msg_len))
+ return error("a NUL byte in commit log message not allowed.");
+
+ nparents = commit_list_count(parents);
+ parent_buf = xcalloc(nparents, sizeof(*parent_buf));
+ for (i = 0; i < nparents; i++) {
+ struct commit *parent = pop_commit(&parents);
+ oidcpy(&parent_buf[i], &parent->object.oid);
+ }
+
+ /* should avoid reallocs for the headers */
+ strbuf_init(&buffer, 8192);
+ strbuf_init(&compat_buffer, 8192);
- if (sign_commit && sign_with_header(&buffer, sign_commit)) {
+ write_commit_tree(&buffer, msg, msg_len, tree, parent_buf, nparents, author, committer, extra);
+ if (sign_commit && sign_commit_to_strbuf(&sig, &buffer, sign_commit)) {
result = -1;
goto out;
}
+ if (r->compat_hash_algo) {
+ struct object_id mapped_tree;
+ struct object_id *mapped_parents = xcalloc(nparents, sizeof(*mapped_parents));
+ if (repo_oid_to_algop(r, tree, r->compat_hash_algo, &mapped_tree)) {
+ result = -1;
+ free(mapped_parents);
+ goto out;
+ }
+ for (i = 0; i < nparents; i++)
+ if (repo_oid_to_algop(r, &parent_buf[i], r->compat_hash_algo, &mapped_parents[i])) {
+ result = -1;
+ free(mapped_parents);
+ goto out;
+ }
+ write_commit_tree(&compat_buffer, msg, msg_len, &mapped_tree,
+ mapped_parents, nparents, author, committer, extra);
- result = write_object_file(buffer.buf, buffer.len, OBJ_COMMIT, ret);
+ if (sign_commit && sign_commit_to_strbuf(&compat_sig, &compat_buffer, sign_commit)) {
+ result = -1;
+ goto out;
+ }
+ }
+
+ if (sign_commit) {
+ struct sig_pairs {
+ struct strbuf *sig;
+ const struct git_hash_algo *algo;
+ } bufs [2] = {
+ { &compat_sig, r->compat_hash_algo },
+ { &sig, r->hash_algo },
+ };
+ int i;
+
+ /*
+ * We write algorithms in the order they were implemented in
+ * Git to produce a stable hash when multiple algorithms are
+ * used.
+ */
+ if (r->compat_hash_algo && hash_algo_by_ptr(bufs[0].algo) > hash_algo_by_ptr(bufs[1].algo))
+ SWAP(bufs[0], bufs[1]);
+
+ /*
+ * We traverse each algorithm in order, and apply the signature
+ * to each buffer.
+ */
+ for (i = 0; i < ARRAY_SIZE(bufs); i++) {
+ if (!bufs[i].algo)
+ continue;
+ add_commit_signature(&buffer, bufs[i].sig, bufs[i].algo);
+ if (r->compat_hash_algo)
+ add_commit_signature(&compat_buffer, bufs[i].sig, bufs[i].algo);
+ }
+ }
+
+ /* And check the encoding. */
+ if (encoding_is_utf8 && (!verify_utf8(&buffer) || !verify_utf8(&compat_buffer)))
+ fprintf(stderr, _(commit_utf8_warn));
+
+ if (r->compat_hash_algo) {
+ hash_object_file(r->compat_hash_algo, compat_buffer.buf, compat_buffer.len,
+ OBJ_COMMIT, &compat_oid_buf);
+ compat_oid = &compat_oid_buf;
+ }
+
+ result = write_object_file_flags(buffer.buf, buffer.len, OBJ_COMMIT,
+ ret, compat_oid, 0);
out:
strbuf_release(&buffer);
+ strbuf_release(&compat_buffer);
+ strbuf_release(&sig);
+ strbuf_release(&compat_sig);
return result;
}
--
2.41.0
^ permalink raw reply related
* [PATCH 07/30] object-file: Update the loose object map when writing loose objects
From: Eric W. Biederman @ 2023-09-27 19:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson, Eric W. Biederman
In-Reply-To: <87jzsbjt0a.fsf@gmail.froward.int.ebiederm.org>
From: "Eric W. Biederman" <ebiederm@xmission.com>
To implement SHA1 compatibility on SHA256 repositories the loose
object map needs to be updated whenver a loose object is written.
Updating the loose object map this way allows git to support
the old hash algorithm in constant time.
The functions write_loose_object, and stream_loose_object are
the only two functions that write to the loose object store.
Update stream_loose_object to compute the compatibiilty hash, update
the loose object, and then call repo_add_loose_object_map to update
the loose object map.
Update write_object_file_flags to convert the object into
it's compatibility encoding, hash the compatibility encoding,
write the object, and then update the loose object map.
Update force_object_loose to lookup the hash of the compatibility
encoding, write the loose object, and then update the loose object
map.
Update write_object_file_literally to convert the object into it's
compatibility hash encoding, hash the compatibility enconding, write
the object, and then update the loose object map, when the type string
is a known type. For objects with an unknown type this results in a
partially broken repository, as the objects are not mapped.
The point of write_object_file_literally is to generate a partially
broken repository for testing. For testing skipping writing the loose
object map is much more useful than refusing to write the broken
object at all.
Except that the loose objects are updated before the loose object map
I have not done any analysis to see how robust this scheme is in the
event of failure.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
---
object-file.c | 113 ++++++++++++++++++++++++++++++++++++++++++--------
1 file changed, 95 insertions(+), 18 deletions(-)
diff --git a/object-file.c b/object-file.c
index 7c7afe579364..4ad31f25c555 100644
--- a/object-file.c
+++ b/object-file.c
@@ -43,6 +43,8 @@
#include "setup.h"
#include "submodule.h"
#include "fsck.h"
+#include "loose.h"
+#include "object-file-convert.h"
/* The maximum size for an object header. */
#define MAX_HEADER_LEN 32
@@ -1952,9 +1954,12 @@ static int start_loose_object_common(struct strbuf *tmp_file,
const char *filename, unsigned flags,
git_zstream *stream,
unsigned char *buf, size_t buflen,
- git_hash_ctx *c,
+ git_hash_ctx *c, git_hash_ctx *compat_c,
char *hdr, int hdrlen)
{
+ struct repository *repo = the_repository;
+ const struct git_hash_algo *algo = repo->hash_algo;
+ const struct git_hash_algo *compat = repo->compat_hash_algo;
int fd;
fd = create_tmpfile(tmp_file, filename);
@@ -1974,14 +1979,18 @@ static int start_loose_object_common(struct strbuf *tmp_file,
git_deflate_init(stream, zlib_compression_level);
stream->next_out = buf;
stream->avail_out = buflen;
- the_hash_algo->init_fn(c);
+ algo->init_fn(c);
+ if (compat && compat_c)
+ compat->init_fn(compat_c);
/* Start to feed header to zlib stream */
stream->next_in = (unsigned char *)hdr;
stream->avail_in = hdrlen;
while (git_deflate(stream, 0) == Z_OK)
; /* nothing */
- the_hash_algo->update_fn(c, hdr, hdrlen);
+ algo->update_fn(c, hdr, hdrlen);
+ if (compat && compat_c)
+ compat->update_fn(compat_c, hdr, hdrlen);
return fd;
}
@@ -1990,16 +1999,21 @@ static int start_loose_object_common(struct strbuf *tmp_file,
* Common steps for the inner git_deflate() loop for writing loose
* objects. Returns what git_deflate() returns.
*/
-static int write_loose_object_common(git_hash_ctx *c,
+static int write_loose_object_common(git_hash_ctx *c, git_hash_ctx *compat_c,
git_zstream *stream, const int flush,
unsigned char *in0, const int fd,
unsigned char *compressed,
const size_t compressed_len)
{
+ struct repository *repo = the_repository;
+ const struct git_hash_algo *algo = repo->hash_algo;
+ const struct git_hash_algo *compat = repo->compat_hash_algo;
int ret;
ret = git_deflate(stream, flush ? Z_FINISH : 0);
- the_hash_algo->update_fn(c, in0, stream->next_in - in0);
+ algo->update_fn(c, in0, stream->next_in - in0);
+ if (compat && compat_c)
+ compat->update_fn(compat_c, in0, stream->next_in - in0);
if (write_in_full(fd, compressed, stream->next_out - compressed) < 0)
die_errno(_("unable to write loose object file"));
stream->next_out = compressed;
@@ -2014,15 +2028,21 @@ static int write_loose_object_common(git_hash_ctx *c,
* - End the compression of zlib stream.
* - Get the calculated oid to "oid".
*/
-static int end_loose_object_common(git_hash_ctx *c, git_zstream *stream,
- struct object_id *oid)
+static int end_loose_object_common(git_hash_ctx *c, git_hash_ctx *compat_c,
+ git_zstream *stream, struct object_id *oid,
+ struct object_id *compat_oid)
{
+ struct repository *repo = the_repository;
+ const struct git_hash_algo *algo = repo->hash_algo;
+ const struct git_hash_algo *compat = repo->compat_hash_algo;
int ret;
ret = git_deflate_end_gently(stream);
if (ret != Z_OK)
return ret;
- the_hash_algo->final_oid_fn(oid, c);
+ algo->final_oid_fn(oid, c);
+ if (compat && compat_c)
+ compat->final_oid_fn(compat_oid, compat_c);
return Z_OK;
}
@@ -2046,7 +2066,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
fd = start_loose_object_common(&tmp_file, filename.buf, flags,
&stream, compressed, sizeof(compressed),
- &c, hdr, hdrlen);
+ &c, NULL, hdr, hdrlen);
if (fd < 0)
return -1;
@@ -2056,14 +2076,14 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
do {
unsigned char *in0 = stream.next_in;
- ret = write_loose_object_common(&c, &stream, 1, in0, fd,
+ ret = write_loose_object_common(&c, NULL, &stream, 1, in0, fd,
compressed, sizeof(compressed));
} while (ret == Z_OK);
if (ret != Z_STREAM_END)
die(_("unable to deflate new object %s (%d)"), oid_to_hex(oid),
ret);
- ret = end_loose_object_common(&c, &stream, ¶no_oid);
+ ret = end_loose_object_common(&c, NULL, &stream, ¶no_oid, NULL);
if (ret != Z_OK)
die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
ret);
@@ -2108,10 +2128,12 @@ static int freshen_packed_object(const struct object_id *oid)
int stream_loose_object(struct input_stream *in_stream, size_t len,
struct object_id *oid)
{
+ const struct git_hash_algo *compat = the_repository->compat_hash_algo;
+ struct object_id compat_oid;
int fd, ret, err = 0, flush = 0;
unsigned char compressed[4096];
git_zstream stream;
- git_hash_ctx c;
+ git_hash_ctx c, compat_c;
struct strbuf tmp_file = STRBUF_INIT;
struct strbuf filename = STRBUF_INIT;
int dirlen;
@@ -2135,7 +2157,7 @@ int stream_loose_object(struct input_stream *in_stream, size_t len,
*/
fd = start_loose_object_common(&tmp_file, filename.buf, 0,
&stream, compressed, sizeof(compressed),
- &c, hdr, hdrlen);
+ &c, &compat_c, hdr, hdrlen);
if (fd < 0) {
err = -1;
goto cleanup;
@@ -2153,7 +2175,7 @@ int stream_loose_object(struct input_stream *in_stream, size_t len,
if (in_stream->is_finished)
flush = 1;
}
- ret = write_loose_object_common(&c, &stream, flush, in0, fd,
+ ret = write_loose_object_common(&c, &compat_c, &stream, flush, in0, fd,
compressed, sizeof(compressed));
/*
* Unlike write_loose_object(), we do not have the entire
@@ -2176,7 +2198,7 @@ int stream_loose_object(struct input_stream *in_stream, size_t len,
*/
if (ret != Z_STREAM_END)
die(_("unable to stream deflate new object (%d)"), ret);
- ret = end_loose_object_common(&c, &stream, oid);
+ ret = end_loose_object_common(&c, &compat_c, &stream, oid, &compat_oid);
if (ret != Z_OK)
die(_("deflateEnd on stream object failed (%d)"), ret);
close_loose_object(fd, tmp_file.buf);
@@ -2203,6 +2225,8 @@ int stream_loose_object(struct input_stream *in_stream, size_t len,
}
err = finalize_object_file(tmp_file.buf, filename.buf);
+ if (!err && compat)
+ err = repo_add_loose_object_map(the_repository, oid, &compat_oid);
cleanup:
strbuf_release(&tmp_file);
strbuf_release(&filename);
@@ -2213,17 +2237,38 @@ int write_object_file_flags(const void *buf, unsigned long len,
enum object_type type, struct object_id *oid,
unsigned flags)
{
+ struct repository *repo = the_repository;
+ const struct git_hash_algo *algo = repo->hash_algo;
+ const struct git_hash_algo *compat = repo->compat_hash_algo;
+ struct object_id compat_oid;
char hdr[MAX_HEADER_LEN];
int hdrlen = sizeof(hdr);
+ /* Generate compat_oid */
+ if (compat) {
+ if (type == OBJ_BLOB)
+ hash_object_file(compat, buf, len, type, &compat_oid);
+ else {
+ struct strbuf converted = STRBUF_INIT;
+ convert_object_file(&converted, algo, compat,
+ buf, len, type, 0);
+ hash_object_file(compat, converted.buf, converted.len,
+ type, &compat_oid);
+ strbuf_release(&converted);
+ }
+ }
+
/* Normally if we have it in the pack then we do not bother writing
* it out into .git/objects/??/?{38} file.
*/
- write_object_file_prepare(the_hash_algo, buf, len, type, oid, hdr,
- &hdrlen);
+ write_object_file_prepare(algo, buf, len, type, oid, hdr, &hdrlen);
if (freshen_packed_object(oid) || freshen_loose_object(oid))
return 0;
- return write_loose_object(oid, hdr, hdrlen, buf, len, 0, flags);
+ if (write_loose_object(oid, hdr, hdrlen, buf, len, 0, flags))
+ return -1;
+ if (compat)
+ return repo_add_loose_object_map(repo, oid, &compat_oid);
+ return 0;
}
int write_object_file_literally(const void *buf, unsigned long len,
@@ -2231,7 +2276,27 @@ int write_object_file_literally(const void *buf, unsigned long len,
unsigned flags)
{
char *header;
+ struct repository *repo = the_repository;
+ const struct git_hash_algo *algo = repo->hash_algo;
+ const struct git_hash_algo *compat = repo->compat_hash_algo;
+ struct object_id compat_oid;
int hdrlen, status = 0;
+ int compat_type = -1;
+
+ if (compat) {
+ compat_type = type_from_string_gently(type, -1, 1);
+ if (compat_type == OBJ_BLOB)
+ hash_object_file(compat, buf, len, compat_type,
+ &compat_oid);
+ else if (compat_type != -1) {
+ struct strbuf converted = STRBUF_INIT;
+ convert_object_file(&converted, algo, compat,
+ buf, len, compat_type, 0);
+ hash_object_file(compat, converted.buf, converted.len,
+ compat_type, &compat_oid);
+ strbuf_release(&converted);
+ }
+ }
/* type string, SP, %lu of the length plus NUL must fit this */
hdrlen = strlen(type) + MAX_HEADER_LEN;
@@ -2244,6 +2309,8 @@ int write_object_file_literally(const void *buf, unsigned long len,
if (freshen_packed_object(oid) || freshen_loose_object(oid))
goto cleanup;
status = write_loose_object(oid, header, hdrlen, buf, len, 0, 0);
+ if (compat_type != -1)
+ return repo_add_loose_object_map(repo, oid, &compat_oid);
cleanup:
free(header);
@@ -2252,9 +2319,12 @@ int write_object_file_literally(const void *buf, unsigned long len,
int force_object_loose(const struct object_id *oid, time_t mtime)
{
+ struct repository *repo = the_repository;
+ const struct git_hash_algo *compat = repo->compat_hash_algo;
void *buf;
unsigned long len;
struct object_info oi = OBJECT_INFO_INIT;
+ struct object_id compat_oid;
enum object_type type;
char hdr[MAX_HEADER_LEN];
int hdrlen;
@@ -2267,8 +2337,15 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
oi.contentp = &buf;
if (oid_object_info_extended(the_repository, oid, &oi, 0))
return error(_("cannot read object for %s"), oid_to_hex(oid));
+ if (compat) {
+ if (repo_oid_to_algop(repo, oid, compat, &compat_oid))
+ return error(_("cannot map object %s to %s"),
+ oid_to_hex(oid), compat->name);
+ }
hdrlen = format_object_header(hdr, sizeof(hdr), type, len);
ret = write_loose_object(oid, hdr, hdrlen, buf, len, mtime, 0);
+ if (!ret && compat)
+ ret = repo_add_loose_object_map(the_repository, oid, &compat_oid);
free(buf);
return ret;
--
2.41.0
^ permalink raw reply related
* [PATCH 08/30] object-file: Add a compat_oid_in parameter to write_object_file_flags
From: Eric W. Biederman @ 2023-09-27 19:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson, Eric W. Biederman
In-Reply-To: <87jzsbjt0a.fsf@gmail.froward.int.ebiederm.org>
From: "Eric W. Biederman" <ebiederm@xmission.com>
To create the proper signatures for commit objects both versions of
the commit object need to be generated and signed. After that it is
a waste to throw away the work of generating the compatibility hash
so update write_object_file_flags to take a compatibility hash input
parameter that it can use to skip the work of generating the
compatability hash.
Update the places that don't generate the compatability hash to
pass NULL so it is easy to tell write_object_file_flags should
not attempt to use their compatability hash.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
---
cache-tree.c | 2 +-
object-file.c | 6 ++++--
object-store-ll.h | 4 ++--
3 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/cache-tree.c b/cache-tree.c
index 641427ed410a..ddc7d3d86959 100644
--- a/cache-tree.c
+++ b/cache-tree.c
@@ -448,7 +448,7 @@ static int update_one(struct cache_tree *it,
hash_object_file(the_hash_algo, buffer.buf, buffer.len,
OBJ_TREE, &it->oid);
} else if (write_object_file_flags(buffer.buf, buffer.len, OBJ_TREE,
- &it->oid, flags & WRITE_TREE_SILENT
+ &it->oid, NULL, flags & WRITE_TREE_SILENT
? HASH_SILENT : 0)) {
strbuf_release(&buffer);
return -1;
diff --git a/object-file.c b/object-file.c
index 4ad31f25c555..d66d11890696 100644
--- a/object-file.c
+++ b/object-file.c
@@ -2235,7 +2235,7 @@ int stream_loose_object(struct input_stream *in_stream, size_t len,
int write_object_file_flags(const void *buf, unsigned long len,
enum object_type type, struct object_id *oid,
- unsigned flags)
+ struct object_id *compat_oid_in, unsigned flags)
{
struct repository *repo = the_repository;
const struct git_hash_algo *algo = repo->hash_algo;
@@ -2246,7 +2246,9 @@ int write_object_file_flags(const void *buf, unsigned long len,
/* Generate compat_oid */
if (compat) {
- if (type == OBJ_BLOB)
+ if (compat_oid_in)
+ oidcpy(&compat_oid, compat_oid_in);
+ else if (type == OBJ_BLOB)
hash_object_file(compat, buf, len, type, &compat_oid);
else {
struct strbuf converted = STRBUF_INIT;
diff --git a/object-store-ll.h b/object-store-ll.h
index bc76d6bec80d..c5f2bb2fc2fe 100644
--- a/object-store-ll.h
+++ b/object-store-ll.h
@@ -255,11 +255,11 @@ void hash_object_file(const struct git_hash_algo *algo, const void *buf,
int write_object_file_flags(const void *buf, unsigned long len,
enum object_type type, struct object_id *oid,
- unsigned flags);
+ struct object_id *comapt_oid_in, unsigned flags);
static inline int write_object_file(const void *buf, unsigned long len,
enum object_type type, struct object_id *oid)
{
- return write_object_file_flags(buf, len, type, oid, 0);
+ return write_object_file_flags(buf, len, type, oid, NULL, 0);
}
int write_object_file_literally(const void *buf, unsigned long len,
--
2.41.0
^ permalink raw reply related
* [PATCH 06/30] loose: Compatibilty short name support
From: Eric W. Biederman @ 2023-09-27 19:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson, Eric W. Biederman
In-Reply-To: <87jzsbjt0a.fsf@gmail.froward.int.ebiederm.org>
From: "Eric W. Biederman" <ebiederm@xmission.com>
Update loose_objects_cache when udpating the loose objects map. This
oidtree is used to discover which oids are possibilities when
resolving short names, and it can support a mixture of sha1
and sha256 oids.
With this any oid recorded objects/loose-objects-idx is usable
for resolving an oid to an object.
To make this maintainable a helper insert_loose_map is factored
out of load_one_loose_object_map and repo_add_loose_object_map,
and then modified to also update the loose_objects_cache.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
---
loose.c | 37 +++++++++++++++++++++++++------------
1 file changed, 25 insertions(+), 12 deletions(-)
diff --git a/loose.c b/loose.c
index 28d11593b2ea..bb50d43cd1d9 100644
--- a/loose.c
+++ b/loose.c
@@ -7,6 +7,7 @@
#include "gettext.h"
#include "loose.h"
#include "lockfile.h"
+#include "oidtree.h"
static const char *loose_object_header = "# loose-object-idx\n";
@@ -42,6 +43,21 @@ static int insert_oid_pair(kh_oid_map_t *map, const struct object_id *key, const
return 1;
}
+static int insert_loose_map(struct object_directory *odb,
+ const struct object_id *oid,
+ const struct object_id *compat_oid)
+{
+ struct loose_object_map *map = odb->loose_map;
+ int inserted = 0;
+
+ inserted |= insert_oid_pair(map->to_compat, oid, compat_oid);
+ inserted |= insert_oid_pair(map->to_storage, compat_oid, oid);
+ if (inserted)
+ oidtree_insert(odb->loose_objects_cache, compat_oid);
+
+ return inserted;
+}
+
static int load_one_loose_object_map(struct repository *repo, struct object_directory *dir)
{
struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
@@ -49,15 +65,14 @@ static int load_one_loose_object_map(struct repository *repo, struct object_dire
if (!dir->loose_map)
loose_object_map_init(&dir->loose_map);
+ if (!dir->loose_objects_cache) {
+ ALLOC_ARRAY(dir->loose_objects_cache, 1);
+ oidtree_init(dir->loose_objects_cache);
+ }
- insert_oid_pair(dir->loose_map->to_compat, repo->hash_algo->empty_tree, repo->compat_hash_algo->empty_tree);
- insert_oid_pair(dir->loose_map->to_storage, repo->compat_hash_algo->empty_tree, repo->hash_algo->empty_tree);
-
- insert_oid_pair(dir->loose_map->to_compat, repo->hash_algo->empty_blob, repo->compat_hash_algo->empty_blob);
- insert_oid_pair(dir->loose_map->to_storage, repo->compat_hash_algo->empty_blob, repo->hash_algo->empty_blob);
-
- insert_oid_pair(dir->loose_map->to_compat, repo->hash_algo->null_oid, repo->compat_hash_algo->null_oid);
- insert_oid_pair(dir->loose_map->to_storage, repo->compat_hash_algo->null_oid, repo->hash_algo->null_oid);
+ insert_loose_map(dir, repo->hash_algo->empty_tree, repo->compat_hash_algo->empty_tree);
+ insert_loose_map(dir, repo->hash_algo->empty_blob, repo->compat_hash_algo->empty_blob);
+ insert_loose_map(dir, repo->hash_algo->null_oid, repo->compat_hash_algo->null_oid);
strbuf_git_common_path(&path, repo, "objects/loose-object-idx");
fp = fopen(path.buf, "rb");
@@ -75,8 +90,7 @@ static int load_one_loose_object_map(struct repository *repo, struct object_dire
parse_oid_hex_algop(p, &compat_oid, &p, repo->compat_hash_algo) ||
p != buf.buf + buf.len)
goto err;
- insert_oid_pair(dir->loose_map->to_compat, &oid, &compat_oid);
- insert_oid_pair(dir->loose_map->to_storage, &compat_oid, &oid);
+ insert_loose_map(dir, &oid, &compat_oid);
}
strbuf_release(&buf);
@@ -195,8 +209,7 @@ int repo_add_loose_object_map(struct repository *repo, const struct object_id *o
if (!should_use_loose_object_map(repo))
return 0;
- inserted |= insert_oid_pair(repo->objects->odb->loose_map->to_compat, oid, compat_oid);
- inserted |= insert_oid_pair(repo->objects->odb->loose_map->to_storage, compat_oid, oid);
+ inserted = insert_loose_map(repo->objects->odb, oid, compat_oid);
if (inserted)
return write_one_object(repo, oid, compat_oid);
return 0;
--
2.41.0
^ permalink raw reply related
* [PATCH 05/30] loose: add a mapping between SHA-1 and SHA-256 for loose objects
From: Eric W. Biederman @ 2023-09-27 19:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson, Eric W . Biederman
In-Reply-To: <87jzsbjt0a.fsf@gmail.froward.int.ebiederm.org>
From: "brian m. carlson" <sandals@crustytoothpaste.net>
As part of the transition plan, we'd like to add a file in the .git
directory that maps loose objects between SHA-1 and SHA-256. Let's
implement the specification in the transition plan and store this data
on a per-repository basis in struct repository.
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
---
Makefile | 1 +
loose.c | 245 ++++++++++++++++++++++++++++++++++++++++++
loose.h | 22 ++++
object-file-convert.c | 14 ++-
object-store-ll.h | 3 +
object.c | 2 +
repository.c | 6 ++
7 files changed, 292 insertions(+), 1 deletion(-)
create mode 100644 loose.c
create mode 100644 loose.h
diff --git a/Makefile b/Makefile
index f7e824f25cda..3c18664def9a 100644
--- a/Makefile
+++ b/Makefile
@@ -1053,6 +1053,7 @@ LIB_OBJS += list-objects-filter.o
LIB_OBJS += list-objects.o
LIB_OBJS += lockfile.o
LIB_OBJS += log-tree.o
+LIB_OBJS += loose.o
LIB_OBJS += ls-refs.o
LIB_OBJS += mailinfo.o
LIB_OBJS += mailmap.o
diff --git a/loose.c b/loose.c
new file mode 100644
index 000000000000..28d11593b2ea
--- /dev/null
+++ b/loose.c
@@ -0,0 +1,245 @@
+#include "git-compat-util.h"
+#include "hash.h"
+#include "path.h"
+#include "object-store.h"
+#include "hex.h"
+#include "wrapper.h"
+#include "gettext.h"
+#include "loose.h"
+#include "lockfile.h"
+
+static const char *loose_object_header = "# loose-object-idx\n";
+
+static inline int should_use_loose_object_map(struct repository *repo)
+{
+ return repo->compat_hash_algo && repo->gitdir;
+}
+
+void loose_object_map_init(struct loose_object_map **map)
+{
+ struct loose_object_map *m;
+ m = xmalloc(sizeof(**map));
+ m->to_compat = kh_init_oid_map();
+ m->to_storage = kh_init_oid_map();
+ *map = m;
+}
+
+static int insert_oid_pair(kh_oid_map_t *map, const struct object_id *key, const struct object_id *value)
+{
+ khiter_t pos;
+ int ret;
+ struct object_id *stored;
+
+ pos = kh_put_oid_map(map, *key, &ret);
+
+ /* This item already exists in the map. */
+ if (ret == 0)
+ return 0;
+
+ stored = xmalloc(sizeof(*stored));
+ oidcpy(stored, value);
+ kh_value(map, pos) = stored;
+ return 1;
+}
+
+static int load_one_loose_object_map(struct repository *repo, struct object_directory *dir)
+{
+ struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
+ FILE *fp;
+
+ if (!dir->loose_map)
+ loose_object_map_init(&dir->loose_map);
+
+ insert_oid_pair(dir->loose_map->to_compat, repo->hash_algo->empty_tree, repo->compat_hash_algo->empty_tree);
+ insert_oid_pair(dir->loose_map->to_storage, repo->compat_hash_algo->empty_tree, repo->hash_algo->empty_tree);
+
+ insert_oid_pair(dir->loose_map->to_compat, repo->hash_algo->empty_blob, repo->compat_hash_algo->empty_blob);
+ insert_oid_pair(dir->loose_map->to_storage, repo->compat_hash_algo->empty_blob, repo->hash_algo->empty_blob);
+
+ insert_oid_pair(dir->loose_map->to_compat, repo->hash_algo->null_oid, repo->compat_hash_algo->null_oid);
+ insert_oid_pair(dir->loose_map->to_storage, repo->compat_hash_algo->null_oid, repo->hash_algo->null_oid);
+
+ strbuf_git_common_path(&path, repo, "objects/loose-object-idx");
+ fp = fopen(path.buf, "rb");
+ if (!fp)
+ return 0;
+
+ errno = 0;
+ if (strbuf_getwholeline(&buf, fp, '\n') || strcmp(buf.buf, loose_object_header))
+ goto err;
+ while (!strbuf_getline_lf(&buf, fp)) {
+ const char *p;
+ struct object_id oid, compat_oid;
+ if (parse_oid_hex_algop(buf.buf, &oid, &p, repo->hash_algo) ||
+ *p++ != ' ' ||
+ parse_oid_hex_algop(p, &compat_oid, &p, repo->compat_hash_algo) ||
+ p != buf.buf + buf.len)
+ goto err;
+ insert_oid_pair(dir->loose_map->to_compat, &oid, &compat_oid);
+ insert_oid_pair(dir->loose_map->to_storage, &compat_oid, &oid);
+ }
+
+ strbuf_release(&buf);
+ strbuf_release(&path);
+ return errno ? -1 : 0;
+err:
+ strbuf_release(&buf);
+ strbuf_release(&path);
+ return -1;
+}
+
+int repo_read_loose_object_map(struct repository *repo)
+{
+ struct object_directory *dir;
+
+ if (!should_use_loose_object_map(repo))
+ return 0;
+
+ prepare_alt_odb(repo);
+
+ for (dir = repo->objects->odb; dir; dir = dir->next) {
+ if (load_one_loose_object_map(repo, dir) < 0) {
+ return -1;
+ }
+ }
+ return 0;
+}
+
+int repo_write_loose_object_map(struct repository *repo)
+{
+ kh_oid_map_t *map = repo->objects->odb->loose_map->to_compat;
+ struct lock_file lock;
+ int fd;
+ khiter_t iter;
+ struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
+
+ if (!should_use_loose_object_map(repo))
+ return 0;
+
+ strbuf_git_common_path(&path, repo, "objects/loose-object-idx");
+ fd = hold_lock_file_for_update_timeout(&lock, path.buf, LOCK_DIE_ON_ERROR, -1);
+ iter = kh_begin(map);
+ if (write_in_full(fd, loose_object_header, strlen(loose_object_header)) < 0)
+ goto errout;
+
+ for (; iter != kh_end(map); iter++) {
+ if (kh_exist(map, iter)) {
+ if (oideq(&kh_key(map, iter), the_hash_algo->empty_tree) ||
+ oideq(&kh_key(map, iter), the_hash_algo->empty_blob))
+ continue;
+ strbuf_addf(&buf, "%s %s\n", oid_to_hex(&kh_key(map, iter)), oid_to_hex(kh_value(map, iter)));
+ if (write_in_full(fd, buf.buf, buf.len) < 0)
+ goto errout;
+ strbuf_reset(&buf);
+ }
+ }
+ strbuf_release(&buf);
+ if (commit_lock_file(&lock) < 0) {
+ error_errno(_("could not write loose object index %s"), path.buf);
+ strbuf_release(&path);
+ return -1;
+ }
+ strbuf_release(&path);
+ return 0;
+errout:
+ rollback_lock_file(&lock);
+ strbuf_release(&buf);
+ error_errno(_("failed to write loose object index %s\n"), path.buf);
+ strbuf_release(&path);
+ return -1;
+}
+
+static int write_one_object(struct repository *repo, const struct object_id *oid,
+ const struct object_id *compat_oid)
+{
+ struct lock_file lock;
+ int fd;
+ struct stat st;
+ struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
+
+ strbuf_git_common_path(&path, repo, "objects/loose-object-idx");
+ hold_lock_file_for_update_timeout(&lock, path.buf, LOCK_DIE_ON_ERROR, -1);
+
+ fd = open(path.buf, O_WRONLY | O_CREAT | O_APPEND, 0666);
+ if (fd < 0)
+ goto errout;
+ if (fstat(fd, &st) < 0)
+ goto errout;
+ if (!st.st_size && write_in_full(fd, loose_object_header, strlen(loose_object_header)) < 0)
+ goto errout;
+
+ strbuf_addf(&buf, "%s %s\n", oid_to_hex(oid), oid_to_hex(compat_oid));
+ if (write_in_full(fd, buf.buf, buf.len) < 0)
+ goto errout;
+ if (close(fd))
+ goto errout;
+ adjust_shared_perm(path.buf);
+ rollback_lock_file(&lock);
+ strbuf_release(&buf);
+ strbuf_release(&path);
+ return 0;
+errout:
+ error_errno(_("failed to write loose object index %s\n"), path.buf);
+ close(fd);
+ rollback_lock_file(&lock);
+ strbuf_release(&buf);
+ strbuf_release(&path);
+ return -1;
+}
+
+int repo_add_loose_object_map(struct repository *repo, const struct object_id *oid,
+ const struct object_id *compat_oid)
+{
+ int inserted = 0;
+
+ if (!should_use_loose_object_map(repo))
+ return 0;
+
+ inserted |= insert_oid_pair(repo->objects->odb->loose_map->to_compat, oid, compat_oid);
+ inserted |= insert_oid_pair(repo->objects->odb->loose_map->to_storage, compat_oid, oid);
+ if (inserted)
+ return write_one_object(repo, oid, compat_oid);
+ return 0;
+}
+
+int repo_loose_object_map_oid(struct repository *repo,
+ const struct object_id *src,
+ const struct git_hash_algo *to,
+ struct object_id *dest)
+
+{
+ struct object_directory *dir;
+ kh_oid_map_t *map;
+ khiter_t pos;
+
+ for (dir = repo->objects->odb; dir; dir = dir->next) {
+ struct loose_object_map *loose_map = dir->loose_map;
+ if (!loose_map)
+ continue;
+ map = (to == repo->compat_hash_algo) ?
+ loose_map->to_compat :
+ loose_map->to_storage;
+ pos = kh_get_oid_map(map, *src);
+ if (pos < kh_end(map)) {
+ oidcpy(dest, kh_value(map, pos));
+ return 0;
+ }
+ }
+ return -1;
+}
+
+void loose_object_map_clear(struct loose_object_map **map)
+{
+ struct loose_object_map *m = *map;
+ struct object_id *oid;
+
+ if (!m)
+ return;
+
+ kh_foreach_value(m->to_compat, oid, free(oid));
+ kh_foreach_value(m->to_storage, oid, free(oid));
+ kh_destroy_oid_map(m->to_compat);
+ kh_destroy_oid_map(m->to_storage);
+ free(m);
+ *map = NULL;
+}
diff --git a/loose.h b/loose.h
new file mode 100644
index 000000000000..2c2957072c5f
--- /dev/null
+++ b/loose.h
@@ -0,0 +1,22 @@
+#ifndef LOOSE_H
+#define LOOSE_H
+
+#include "khash.h"
+
+struct loose_object_map {
+ kh_oid_map_t *to_compat;
+ kh_oid_map_t *to_storage;
+};
+
+void loose_object_map_init(struct loose_object_map **map);
+void loose_object_map_clear(struct loose_object_map **map);
+int repo_loose_object_map_oid(struct repository *repo,
+ const struct object_id *src,
+ const struct git_hash_algo *dest_algo,
+ struct object_id *dest);
+int repo_add_loose_object_map(struct repository *repo, const struct object_id *oid,
+ const struct object_id *compat_oid);
+int repo_read_loose_object_map(struct repository *repo);
+int repo_write_loose_object_map(struct repository *repo);
+
+#endif
diff --git a/object-file-convert.c b/object-file-convert.c
index ba3e18f6af44..4d62ed192bf0 100644
--- a/object-file-convert.c
+++ b/object-file-convert.c
@@ -4,6 +4,7 @@
#include "repository.h"
#include "hash-ll.h"
#include "object.h"
+#include "loose.h"
#include "object-file-convert.h"
int repo_oid_to_algop(struct repository *repo, const struct object_id *src,
@@ -21,7 +22,18 @@ int repo_oid_to_algop(struct repository *repo, const struct object_id *src,
oidcpy(dest, src);
return 0;
}
- return -1;
+ if (repo_loose_object_map_oid(repo, src, to, dest)) {
+ /*
+ * We may have loaded the object map at repo initialization but
+ * another process (perhaps upstream of a pipe from us) may have
+ * written a new object into the map. If the object is missing,
+ * let's reload the map to see if the object has appeared.
+ */
+ repo_read_loose_object_map(repo);
+ if (repo_loose_object_map_oid(repo, src, to, dest))
+ return -1;
+ }
+ return 0;
}
int convert_object_file(struct strbuf *outbuf,
diff --git a/object-store-ll.h b/object-store-ll.h
index 26a3895c821c..bc76d6bec80d 100644
--- a/object-store-ll.h
+++ b/object-store-ll.h
@@ -26,6 +26,9 @@ struct object_directory {
uint32_t loose_objects_subdir_seen[8]; /* 256 bits */
struct oidtree *loose_objects_cache;
+ /* Map between object IDs for loose objects. */
+ struct loose_object_map *loose_map;
+
/*
* This is a temporary object store created by the tmp_objdir
* facility. Disable ref updates since the objects in the store
diff --git a/object.c b/object.c
index 2c61e4c86217..186a0a47c0fb 100644
--- a/object.c
+++ b/object.c
@@ -13,6 +13,7 @@
#include "alloc.h"
#include "packfile.h"
#include "commit-graph.h"
+#include "loose.h"
unsigned int get_max_object_index(void)
{
@@ -540,6 +541,7 @@ void free_object_directory(struct object_directory *odb)
{
free(odb->path);
odb_clear_loose_cache(odb);
+ loose_object_map_clear(&odb->loose_map);
free(odb);
}
diff --git a/repository.c b/repository.c
index 80252b79e93e..6214f61cf4e7 100644
--- a/repository.c
+++ b/repository.c
@@ -14,6 +14,7 @@
#include "read-cache-ll.h"
#include "remote.h"
#include "setup.h"
+#include "loose.h"
#include "submodule-config.h"
#include "sparse-index.h"
#include "trace2.h"
@@ -109,6 +110,8 @@ void repo_set_compat_hash_algo(struct repository *repo, int algo)
if (hash_algo_by_ptr(repo->hash_algo) == algo)
BUG("hash_algo and compat_hash_algo match");
repo->compat_hash_algo = algo ? &hash_algos[algo] : NULL;
+ if (repo->compat_hash_algo)
+ repo_read_loose_object_map(repo);
}
/*
@@ -201,6 +204,9 @@ int repo_init(struct repository *repo,
if (worktree)
repo_set_worktree(repo, worktree);
+ if (repo->compat_hash_algo)
+ repo_read_loose_object_map(repo);
+
clear_repository_format(&format);
return 0;
--
2.41.0
^ permalink raw reply related
* [PATCH 03/30] object-names: Support input of oids in any supported hash
From: Eric W. Biederman @ 2023-09-27 19:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson, Eric W. Biederman
In-Reply-To: <87jzsbjt0a.fsf@gmail.froward.int.ebiederm.org>
From: "Eric W. Biederman" <ebiederm@xmission.com>
Support short oids encoded in any algorithm, while ensuring enough of
the oid is specified to disambiguate between all of the oids in the
repository encoded in any algorithm.
By default have the code continue to only accept oids specified in the
storage hash algorithm of the repository, but when something is
ambiguous display all of the possible oids from any oid encoding.
A new flag is added GET_OID_HASH_ANY that when supplied causes the
code to accept oids specified in any hash algorithm, and to return the
oids that were resolved.
This implements the functionality that allows both SHA-1 and SHA-256
object names, from the "Object names on the command line" section of
the hash function transition document.
Care is taken in get_short_oid so that when the result is ambiguous
the output remains the same of GIT_OID_HASH_ANY was not supplied.
If GET_OID_HASH_ANY was supplied objects of any hash algorithm
that match the prefix are displayed.
This required updating repo_for_each_abbrev to give it a parameter
so that it knows to look at all hash algorithms.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
---
builtin/rev-parse.c | 2 +-
hash-ll.h | 1 +
object-name.c | 49 +++++++++++++++++++++++++++++++++++----------
object-name.h | 3 ++-
4 files changed, 42 insertions(+), 13 deletions(-)
diff --git a/builtin/rev-parse.c b/builtin/rev-parse.c
index fde8861ca4e0..43e96765400c 100644
--- a/builtin/rev-parse.c
+++ b/builtin/rev-parse.c
@@ -882,7 +882,7 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)
continue;
}
if (skip_prefix(arg, "--disambiguate=", &arg)) {
- repo_for_each_abbrev(the_repository, arg,
+ repo_for_each_abbrev(the_repository, arg, the_hash_algo,
show_abbrev, NULL);
continue;
}
diff --git a/hash-ll.h b/hash-ll.h
index 10d84cc20888..2cfde63ae1cf 100644
--- a/hash-ll.h
+++ b/hash-ll.h
@@ -145,6 +145,7 @@ struct object_id {
#define GET_OID_RECORD_PATH 0200
#define GET_OID_ONLY_TO_DIE 04000
#define GET_OID_REQUIRE_PATH 010000
+#define GET_OID_HASH_ANY 020000
#define GET_OID_DISAMBIGUATORS \
(GET_OID_COMMIT | GET_OID_COMMITTISH | \
diff --git a/object-name.c b/object-name.c
index 0bfa29dbbfe9..976b7106821b 100644
--- a/object-name.c
+++ b/object-name.c
@@ -25,6 +25,7 @@
#include "midx.h"
#include "commit-reach.h"
#include "date.h"
+#include "object-file-convert.h"
static int get_oid_oneline(struct repository *r, const char *, struct object_id *, struct commit_list *);
@@ -49,6 +50,7 @@ struct disambiguate_state {
static void update_candidates(struct disambiguate_state *ds, const struct object_id *current)
{
+ /* The hash algorithm of the current has already been filtered */
if (ds->always_call_fn) {
ds->ambiguous = ds->fn(ds->repo, current, ds->cb_data) ? 1 : 0;
return;
@@ -134,6 +136,8 @@ static void unique_in_midx(struct multi_pack_index *m,
{
uint32_t num, i, first = 0;
const struct object_id *current = NULL;
+ int len = ds->len > ds->repo->hash_algo->hexsz ?
+ ds->repo->hash_algo->hexsz : ds->len;
num = m->num_objects;
if (!num)
@@ -149,7 +153,7 @@ static void unique_in_midx(struct multi_pack_index *m,
for (i = first; i < num && !ds->ambiguous; i++) {
struct object_id oid;
current = nth_midxed_object_oid(&oid, m, i);
- if (!match_hash(ds->len, ds->bin_pfx.hash, current->hash))
+ if (!match_hash(len, ds->bin_pfx.hash, current->hash))
break;
update_candidates(ds, current);
}
@@ -159,6 +163,8 @@ static void unique_in_pack(struct packed_git *p,
struct disambiguate_state *ds)
{
uint32_t num, i, first = 0;
+ int len = ds->len > ds->repo->hash_algo->hexsz ?
+ ds->repo->hash_algo->hexsz : ds->len;
if (p->multi_pack_index)
return;
@@ -177,7 +183,7 @@ static void unique_in_pack(struct packed_git *p,
for (i = first; i < num && !ds->ambiguous; i++) {
struct object_id oid;
nth_packed_object_id(&oid, p, i);
- if (!match_hash(ds->len, ds->bin_pfx.hash, oid.hash))
+ if (!match_hash(len, ds->bin_pfx.hash, oid.hash))
break;
update_candidates(ds, &oid);
}
@@ -188,6 +194,10 @@ static void find_short_packed_object(struct disambiguate_state *ds)
struct multi_pack_index *m;
struct packed_git *p;
+ /* Skip, unless oids from the storage hash algorithm are wanted */
+ if (ds->bin_pfx.algo && (&hash_algos[ds->bin_pfx.algo] != ds->repo->hash_algo))
+ return;
+
for (m = get_multi_pack_index(ds->repo); m && !ds->ambiguous;
m = m->next)
unique_in_midx(m, ds);
@@ -326,11 +336,12 @@ int set_disambiguate_hint_config(const char *var, const char *value)
static int init_object_disambiguation(struct repository *r,
const char *name, int len,
+ const struct git_hash_algo *algo,
struct disambiguate_state *ds)
{
int i;
- if (len < MINIMUM_ABBREV || len > the_hash_algo->hexsz)
+ if (len < MINIMUM_ABBREV || len > GIT_MAX_HEXSZ)
return -1;
memset(ds, 0, sizeof(*ds));
@@ -357,6 +368,7 @@ static int init_object_disambiguation(struct repository *r,
ds->len = len;
ds->hex_pfx[len] = '\0';
ds->repo = r;
+ ds->bin_pfx.algo = algo ? hash_algo_by_ptr(algo) : GIT_HASH_UNKNOWN;
prepare_alt_odb(r);
return 0;
}
@@ -491,9 +503,10 @@ static int repo_collect_ambiguous(struct repository *r UNUSED,
return collect_ambiguous(oid, data);
}
-static int sort_ambiguous(const void *a, const void *b, void *ctx)
+static int sort_ambiguous(const void *va, const void *vb, void *ctx)
{
struct repository *sort_ambiguous_repo = ctx;
+ const struct object_id *a = va, *b = vb;
int a_type = oid_object_info(sort_ambiguous_repo, a, NULL);
int b_type = oid_object_info(sort_ambiguous_repo, b, NULL);
int a_type_sort;
@@ -503,8 +516,13 @@ static int sort_ambiguous(const void *a, const void *b, void *ctx)
* Sorts by hash within the same object type, just as
* oid_array_for_each_unique() would do.
*/
- if (a_type == b_type)
- return oidcmp(a, b);
+ if (a_type == b_type) {
+ /* Is the hash algorithm the same? */
+ if (a->algo == b->algo)
+ return oidcmp(a, b);
+ else
+ return a->algo > b->algo ? 1 : -1;
+ }
/*
* Between object types show tags, then commits, and finally
@@ -533,8 +551,12 @@ static enum get_oid_result get_short_oid(struct repository *r,
int status;
struct disambiguate_state ds;
int quietly = !!(flags & GET_OID_QUIETLY);
+ const struct git_hash_algo *algo = r->hash_algo;
+
+ if (flags & GET_OID_HASH_ANY)
+ algo = NULL;
- if (init_object_disambiguation(r, name, len, &ds) < 0)
+ if (init_object_disambiguation(r, name, len, algo, &ds) < 0)
return -1;
if (HAS_MULTI_BITS(flags & GET_OID_DISAMBIGUATORS))
@@ -553,6 +575,7 @@ static enum get_oid_result get_short_oid(struct repository *r,
else
ds.fn = default_disambiguate_hint;
+
find_short_object_filename(&ds);
find_short_packed_object(&ds);
status = finish_object_disambiguation(&ds, oid);
@@ -588,7 +611,7 @@ static enum get_oid_result get_short_oid(struct repository *r,
if (!ds.ambiguous)
ds.fn = NULL;
- repo_for_each_abbrev(r, ds.hex_pfx, collect_ambiguous, &collect);
+ repo_for_each_abbrev(r, ds.hex_pfx, algo, collect_ambiguous, &collect);
sort_ambiguous_oid_array(r, &collect);
if (oid_array_for_each(&collect, show_ambiguous_object, &out))
@@ -610,15 +633,17 @@ static enum get_oid_result get_short_oid(struct repository *r,
}
int repo_for_each_abbrev(struct repository *r, const char *prefix,
+ const struct git_hash_algo *algo,
each_abbrev_fn fn, void *cb_data)
{
struct oid_array collect = OID_ARRAY_INIT;
struct disambiguate_state ds;
int ret;
- if (init_object_disambiguation(r, prefix, strlen(prefix), &ds) < 0)
+ if (init_object_disambiguation(r, prefix, strlen(prefix), algo, &ds) < 0)
return -1;
+ ds.bin_pfx.algo = GIT_HASH_UNKNOWN;
ds.always_call_fn = 1;
ds.fn = repo_collect_ambiguous;
ds.cb_data = &collect;
@@ -787,10 +812,12 @@ void strbuf_add_unique_abbrev(struct strbuf *sb, const struct object_id *oid,
int repo_find_unique_abbrev_r(struct repository *r, char *hex,
const struct object_id *oid, int len)
{
+ const struct git_hash_algo *algo =
+ oid->algo ? &hash_algos[oid->algo] : r->hash_algo;
struct disambiguate_state ds;
struct min_abbrev_data mad;
struct object_id oid_ret;
- const unsigned hexsz = r->hash_algo->hexsz;
+ const unsigned hexsz = algo->hexsz;
if (len < 0) {
unsigned long count = repo_approximate_object_count(r);
@@ -826,7 +853,7 @@ int repo_find_unique_abbrev_r(struct repository *r, char *hex,
find_abbrev_len_packed(&mad);
- if (init_object_disambiguation(r, hex, mad.cur_len, &ds) < 0)
+ if (init_object_disambiguation(r, hex, mad.cur_len, algo, &ds) < 0)
return -1;
ds.fn = repo_extend_abbrev_len;
diff --git a/object-name.h b/object-name.h
index 9ae522307148..064ddc97d1fe 100644
--- a/object-name.h
+++ b/object-name.h
@@ -67,7 +67,8 @@ enum get_oid_result get_oid_with_context(struct repository *repo, const char *st
typedef int each_abbrev_fn(const struct object_id *oid, void *);
-int repo_for_each_abbrev(struct repository *r, const char *prefix, each_abbrev_fn, void *);
+int repo_for_each_abbrev(struct repository *r, const char *prefix,
+ const struct git_hash_algo *algo, each_abbrev_fn, void *);
int set_disambiguate_hint_config(const char *var, const char *value);
--
2.41.0
^ permalink raw reply related
* [PATCH 04/30] repository: add a compatibility hash algorithm
From: Eric W. Biederman @ 2023-09-27 19:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson, Eric W. Biederman
In-Reply-To: <87jzsbjt0a.fsf@gmail.froward.int.ebiederm.org>
From: "Eric W. Biederman" <ebiederm@xmission.com>
We currently have support for using a full stage 4 SHA-256
implementation. However, we'd like to support interoperability with
SHA-1 repositories as well. The transition plan anticipates a
compatibility hash algorithm configuration option that we can use to
implement support for this. Let's add an element to the repository
structure that indicates the compatibility hash algorithm so we can use
it when we need to consider interoperability between algorithms.
Add a helper function repo_set_compat_hash_algo that takes a
compatibility hash algorithm and sets "repo->compat_hash_algo". If
GIT_HASH_UNKNOWN is passed as the compatibilty hash algorithm
"repo->compat_hash_algo" is set to NULL.
For now, the code always results in "repo->compat_hash_algo" always
being set to NULL, but that will change once a configuration option
is added.
Inspired-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Eric W. Biederman <ebiederm@xmission.com>
---
repository.c | 8 ++++++++
repository.h | 4 ++++
setup.c | 3 +++
3 files changed, 15 insertions(+)
diff --git a/repository.c b/repository.c
index a7679ceeaa45..80252b79e93e 100644
--- a/repository.c
+++ b/repository.c
@@ -104,6 +104,13 @@ void repo_set_hash_algo(struct repository *repo, int hash_algo)
repo->hash_algo = &hash_algos[hash_algo];
}
+void repo_set_compat_hash_algo(struct repository *repo, int algo)
+{
+ if (hash_algo_by_ptr(repo->hash_algo) == algo)
+ BUG("hash_algo and compat_hash_algo match");
+ repo->compat_hash_algo = algo ? &hash_algos[algo] : NULL;
+}
+
/*
* Attempt to resolve and set the provided 'gitdir' for repository 'repo'.
* Return 0 upon success and a non-zero value upon failure.
@@ -184,6 +191,7 @@ int repo_init(struct repository *repo,
goto error;
repo_set_hash_algo(repo, format.hash_algo);
+ repo_set_compat_hash_algo(repo, GIT_HASH_UNKNOWN);
repo->repository_format_worktree_config = format.worktree_config;
/* take ownership of format.partial_clone */
diff --git a/repository.h b/repository.h
index 5f18486f6465..bf3fc601cc53 100644
--- a/repository.h
+++ b/repository.h
@@ -160,6 +160,9 @@ struct repository {
/* Repository's current hash algorithm, as serialized on disk. */
const struct git_hash_algo *hash_algo;
+ /* Repository's compatibility hash algorithm. */
+ const struct git_hash_algo *compat_hash_algo;
+
/* A unique-id for tracing purposes. */
int trace2_repo_id;
@@ -199,6 +202,7 @@ void repo_set_gitdir(struct repository *repo, const char *root,
const struct set_gitdir_args *extra_args);
void repo_set_worktree(struct repository *repo, const char *path);
void repo_set_hash_algo(struct repository *repo, int algo);
+void repo_set_compat_hash_algo(struct repository *repo, int compat_algo);
void initialize_the_repository(void);
RESULT_MUST_BE_USED
int repo_init(struct repository *r, const char *gitdir, const char *worktree);
diff --git a/setup.c b/setup.c
index ef9f79b8885e..deb5a33fe9e1 100644
--- a/setup.c
+++ b/setup.c
@@ -1572,6 +1572,8 @@ const char *setup_git_directory_gently(int *nongit_ok)
}
if (startup_info->have_repository) {
repo_set_hash_algo(the_repository, repo_fmt.hash_algo);
+ repo_set_compat_hash_algo(the_repository,
+ GIT_HASH_UNKNOWN);
the_repository->repository_format_worktree_config =
repo_fmt.worktree_config;
/* take ownership of repo_fmt.partial_clone */
@@ -1665,6 +1667,7 @@ void check_repository_format(struct repository_format *fmt)
check_repository_format_gently(get_git_dir(), fmt, NULL);
startup_info->have_repository = 1;
repo_set_hash_algo(the_repository, fmt->hash_algo);
+ repo_set_compat_hash_algo(the_repository, GIT_HASH_UNKNOWN);
the_repository->repository_format_worktree_config =
fmt->worktree_config;
the_repository->repository_format_partial_clone =
--
2.41.0
^ permalink raw reply related
* [PATCH 02/30] oid-array: Teach oid-array to handle multiple kinds of oids
From: Eric W. Biederman @ 2023-09-27 19:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson, Eric W. Biederman
In-Reply-To: <87jzsbjt0a.fsf@gmail.froward.int.ebiederm.org>
From: "Eric W. Biederman" <ebiederm@xmission.com>
While looking at how to handle input of both SHA-1 and SHA-256 oids in
get_oid_with_context, I realized that the oid_array in
repo_for_each_abbrev might have more than one kind of oid stored in it
simulataneously.
Update to oid_array_append to ensure that oids added to an oid array
always have an algorithm set.
Update void_hashcmp to first verify two oids use the same hash algorithm
before comparing them to each other.
With that oid-array should be safe to use with differnt kinds of
oids simultaneously.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
---
oid-array.c | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/oid-array.c b/oid-array.c
index 8e4717746c31..1f36651754ed 100644
--- a/oid-array.c
+++ b/oid-array.c
@@ -6,12 +6,20 @@ void oid_array_append(struct oid_array *array, const struct object_id *oid)
{
ALLOC_GROW(array->oid, array->nr + 1, array->alloc);
oidcpy(&array->oid[array->nr++], oid);
+ if (!oid->algo)
+ oid_set_algo(&array->oid[array->nr - 1], the_hash_algo);
array->sorted = 0;
}
-static int void_hashcmp(const void *a, const void *b)
+static int void_hashcmp(const void *va, const void *vb)
{
- return oidcmp(a, b);
+ const struct object_id *a = va, *b = vb;
+ int ret;
+ if (a->algo == b->algo)
+ ret = oidcmp(a, b);
+ else
+ ret = a->algo > b->algo ? 1 : -1;
+ return ret;
}
void oid_array_sort(struct oid_array *array)
--
2.41.0
^ permalink raw reply related
* [PATCH 01/30] object-file-convert: Stubs for converting from one object format to another
From: Eric W. Biederman @ 2023-09-27 19:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson, Eric W. Biederman
In-Reply-To: <87jzsbjt0a.fsf@gmail.froward.int.ebiederm.org>
From: "Eric W. Biederman" <ebiederm@xmission.com>
Two basic functions are provided:
- convert_object_file Takes an object file it's type and hash algorithm
and converts it into the equivalent object file that would
have been generated with hash algorithm "to".
For blob objects there is no converstion to be done and it is an
error to use this function on them.
For commit, tree, and tag objects embedded oids are replaced by the
oids of the objects they refer to with those objects and their
object ids reencoded in with the hash algorithm "to". Signatures
are rearranged so that they remain valid after the object has
been reencoded.
- repo_oid_to_algop which takes an oid that refers to an object file
and returns the oid of the equavalent object file generated
with the target hash algorithm.
The pair of files object-file-convert.c and object-file-convert.h are
introduced to hold as much of this logic as possible to keep this
conversion logic cleanly separated from everything else and in the
hopes that someday the code will be clean enough git can support
compiling out support for sha1 and the various conversion functions.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
---
Makefile | 1 +
object-file-convert.c | 57 +++++++++++++++++++++++++++++++++++++++++++
object-file-convert.h | 24 ++++++++++++++++++
3 files changed, 82 insertions(+)
create mode 100644 object-file-convert.c
create mode 100644 object-file-convert.h
diff --git a/Makefile b/Makefile
index 577630936535..f7e824f25cda 100644
--- a/Makefile
+++ b/Makefile
@@ -1073,6 +1073,7 @@ LIB_OBJS += notes-cache.o
LIB_OBJS += notes-merge.o
LIB_OBJS += notes-utils.o
LIB_OBJS += notes.o
+LIB_OBJS += object-file-convert.o
LIB_OBJS += object-file.o
LIB_OBJS += object-name.o
LIB_OBJS += object.o
diff --git a/object-file-convert.c b/object-file-convert.c
new file mode 100644
index 000000000000..ba3e18f6af44
--- /dev/null
+++ b/object-file-convert.c
@@ -0,0 +1,57 @@
+#include "git-compat-util.h"
+#include "gettext.h"
+#include "strbuf.h"
+#include "repository.h"
+#include "hash-ll.h"
+#include "object.h"
+#include "object-file-convert.h"
+
+int repo_oid_to_algop(struct repository *repo, const struct object_id *src,
+ const struct git_hash_algo *to, struct object_id *dest)
+{
+ /*
+ * If the source alogirthm is not set, then we're using the
+ * default hash algorithm for that object.
+ */
+ const struct git_hash_algo *from =
+ src->algo ? &hash_algos[src->algo] : repo->hash_algo;
+
+ if (from == to) {
+ if (src != dest)
+ oidcpy(dest, src);
+ return 0;
+ }
+ return -1;
+}
+
+int convert_object_file(struct strbuf *outbuf,
+ const struct git_hash_algo *from,
+ const struct git_hash_algo *to,
+ const void *buf, size_t len,
+ enum object_type type,
+ int gentle)
+{
+ int ret;
+
+ /* Don't call this function when no conversion is necessary */
+ if ((from == to) || (type == OBJ_BLOB))
+ die("Refusing noop object file conversion");
+
+ switch (type) {
+ case OBJ_COMMIT:
+ case OBJ_TREE:
+ case OBJ_TAG:
+ default:
+ /* Not implemented yet, so fail. */
+ ret = -1;
+ break;
+ }
+ if (!ret)
+ return 0;
+ if (gentle) {
+ strbuf_release(outbuf);
+ return ret;
+ }
+ die(_("Failed to convert object from %s to %s"),
+ from->name, to->name);
+}
diff --git a/object-file-convert.h b/object-file-convert.h
new file mode 100644
index 000000000000..a4f802aa8eea
--- /dev/null
+++ b/object-file-convert.h
@@ -0,0 +1,24 @@
+#ifndef OBJECT_CONVERT_H
+#define OBJECT_CONVERT_H
+
+struct repository;
+struct object_id;
+struct git_hash_algo;
+struct strbuf;
+#include "object.h"
+
+int repo_oid_to_algop(struct repository *repo, const struct object_id *src,
+ const struct git_hash_algo *to, struct object_id *dest);
+
+/*
+ * Convert an object file from one hash algorithm to another algorithm.
+ * Return -1 on failure, 0 on success.
+ */
+int convert_object_file(struct strbuf *outbuf,
+ const struct git_hash_algo *from,
+ const struct git_hash_algo *to,
+ const void *buf, size_t len,
+ enum object_type type,
+ int gentle);
+
+#endif /* OBJECT_CONVERT_H */
--
2.41.0
^ permalink raw reply related
* [PATCH 00/30] Initial support for multiple hash functions
From: Eric W. Biederman @ 2023-09-27 19:49 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson
I have been going over and over this patchset trying to figure
out if it is ready to be merged. I don't know of any deficiencies
so it is at a point it could benefit from a set of eyes that
are not mine.
I had planned to wait a little bit longer but there are some on-going
conversations that could benefit from people seeing what it means for a
repository to support two hash functions at the same time.
A key part of the hash function transition plan is a way that a single
git repository can inter-operate with git repositories whose storage
hash function is SHA-1 and git repositories whose storage hash function
is SHA-256.
This interoperability can defined in terms of two repositories one whose
storage hash function is SHA-1 and another whose storage hash function
is SHA-256. Those two repositories receive exactly the same objects,
but they store them in different but equivalent ways.
For a repository that has one storage hash function to inter-operate
with a repository that has a different storage hash function requires
the first repository to be able produce it's objects as if they were
stored in the second hash function.
This series of changes focuses on implementing the pieces that allow
a repository that uses one storage hash function to produce the objects
that would have been stored with a second storage hash function.
The final patch in this series is the addition of a test that creates
two repositories one that uses SHA-1 as it's storage hash function
and the other that uses SHA-256 as it's storage hash function.
Identical operations are performed on the two repositories, and their
compatibility objects are compared to verify they are the same.
AKA the SHA-1 repository on the fly generates the objects store in
the SHA-256 repository, and the SHA-256 repository on the fly generates
the objects that are stored in the SHA-1 repository.
There are two fundamental technologies for enabling this.
- The ability to convert a stored object into the object the
other repository would have stored.
- The ability to remember a mapping between SHA-1 and SHA-256 oids
of equivalent objects.
With such technologies it is very easy to implement user facing changes.
To avoid locking git into poor decisions by accident I have done my best
to minimize the user facing changes, while still building the internal
infrastructure that is needed for interoperability.
All of this work is inspired by earlier work on interoperability by
"brian m. carlson" and some of the key pieces of code are still his.
To get to the point where I can test if a SHA-1 and a SHA-256 repository
can on the fly generate each other, I have made some small user-facing
changes.
git rev-parse now supports --output-object-format as a way to query
the internal mapping tables between oids and report the equivalent
oid of the other format.
git cat-file when given a oid that does not match the repositories
storage format will now attempt to find the oids equivalent object that
is stored in the repository and if found dynamically generate the object
that would have been stored in a repository with a different storage
hash function and display the object.
An additional file loose-object-index will be stored in ".git/objects/".
An additional option "extensions.compatObjectFormat" is implemented,
that generates and stores mappings between the oids of objects stored in
the repository and oids of the equivalent objects that would be stored
in a repository show storage format was extensions.compatObjectFormat.
Eric W. Biederman (23):
object-file-convert: Stubs for converting from one object format to another
oid-array: Teach oid-array to handle multiple kinds of oids
object-names: Support input of oids in any supported hash
repository: add a compatibility hash algorithm
loose: Compatibilty short name support
object-file: Update the loose object map when writing loose objects
object-file: Add a compat_oid_in parameter to write_object_file_flags
commit: Convert mergetag before computing the signature of a commit
commit: Export add_header_signature to support handling signatures on tags
tag: sign both hashes
object: Factor out parse_mode out of fast-import and tree-walk into in object.h
object-file-convert: Don't leak when converting tag objects
object-file-convert: Convert commits that embed signed tags
object-file: Update object_info_extended to reencode objects
rev-parse: Add an --output-object-format parameter
builtin/cat-file: Let the oid determine the output algorithm
tree-walk: init_tree_desc take an oid to get the hash algorithm
object-file: Handle compat objects in check_object_signature
builtin/ls-tree: Let the oid determine the output algorithm
test-lib: Compute the compatibility hash so tests may use it
t1006: Rename sha1 to oid
t1006: Test oid compatibility with cat-file
t1016-compatObjectFormat: Add tests to verify the conversion between objects
brian m. carlson (7):
loose: add a mapping between SHA-1 and SHA-256 for loose objects
commit: write commits for both hashes
cache: add a function to read an OID of a specific algorithm
object-file-convert: add a function to convert trees between algorithms
object-file-convert: convert tag objects when writing
object-file-convert: convert commit objects when writing
repository: Implement extensions.compatObjectFormat
Documentation/config/extensions.txt | 12 ++
Documentation/git-rev-parse.txt | 12 ++
Makefile | 3 +
archive.c | 3 +-
builtin/am.c | 6 +-
builtin/cat-file.c | 12 +-
builtin/checkout.c | 8 +-
builtin/clone.c | 2 +-
builtin/commit.c | 2 +-
builtin/fast-import.c | 18 +-
builtin/grep.c | 8 +-
builtin/ls-tree.c | 5 +-
builtin/merge.c | 3 +-
builtin/pack-objects.c | 6 +-
builtin/read-tree.c | 2 +-
builtin/rev-parse.c | 25 ++-
builtin/stash.c | 5 +-
builtin/tag.c | 45 ++++-
cache-tree.c | 4 +-
commit.c | 219 ++++++++++++++++-----
commit.h | 1 +
delta-islands.c | 2 +-
diff-lib.c | 2 +-
fsck.c | 6 +-
hash-ll.h | 1 +
hash.h | 9 +-
http-push.c | 2 +-
list-objects.c | 2 +-
loose.c | 258 ++++++++++++++++++++++++
loose.h | 22 +++
match-trees.c | 4 +-
merge-ort.c | 11 +-
merge-recursive.c | 2 +-
merge.c | 3 +-
object-file-convert.c | 277 ++++++++++++++++++++++++++
object-file-convert.h | 24 +++
object-file.c | 212 ++++++++++++++++++--
object-name.c | 49 +++--
object-name.h | 3 +-
object-store-ll.h | 7 +-
object.c | 2 +
object.h | 18 ++
oid-array.c | 12 +-
pack-bitmap-write.c | 2 +-
packfile.c | 3 +-
reflog.c | 2 +-
repository.c | 14 ++
repository.h | 4 +
revision.c | 4 +-
setup.c | 22 +++
setup.h | 1 +
t/helper/test-delete-gpgsig.c | 62 ++++++
t/helper/test-tool.c | 1 +
t/helper/test-tool.h | 1 +
t/t1006-cat-file.sh | 379 +++++++++++++++++++++---------------
t/t1016-compatObjectFormat.sh | 280 ++++++++++++++++++++++++++
t/t1016/gpg | 2 +
t/test-lib-functions.sh | 17 +-
tree-walk.c | 58 +++---
tree-walk.h | 7 +-
tree.c | 2 +-
walker.c | 2 +-
62 files changed, 1843 insertions(+), 349 deletions(-)
Eric
^ permalink raw reply
* Re: [PATCH v4] revision: add `--ignore-missing-links` user option
From: Karthik Nayak @ 2023-09-27 16:26 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, me
In-Reply-To: <xmqq5y3yw5qz.fsf@gitster.g>
On Mon, Sep 25, 2023 at 6:57 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Karthik Nayak <karthik.188@gmail.com> writes:
>
> > Let me prefix with saying that I was partly wrong. `--missing` does work for
> > trees, only that it's ineffective when used along with the
> > `ignore_missing_links` bit.
> >
> > But for commits, `--missing` was never configured to work with. I
> > did a quick look at the code, we can do something like this for
> > commits too, i.e. add support for the `--missing` option. We'll
> > have to add a new flag (maybe MISSING) so it can be set during
> > within `repo_parse_commit_gently` so we can parse this as a
> > missing object in rev-list.c and act accordingly.
>
> Do you mean that process_parents() would now throw such a commit to
> the resulting list successfully instead of omitting when "--missing"
> is requested? That sounds like a right thing to do but at the same
> time is a fix with major impact.
Yes, but with appropriate flag added. Which will be a new flag.
> I do not offhand know what the
> ramifications are, for example, when bitmap traversal is in use (I
> assume such a missing commit would not be catalogued in the bitmap?).
>
If there is a missing commit or object, will there even be a bitmap?
I can think of the two scenarios:
1. Object is missing before bitmap creation: In such a scenario, the bitmap
doesn't get created, since an object is missing. Could be any type of object.
2. Object is missing after bitmap creation: In this case, the bitmap
already exists
and rev-list won't even know that the commit is missing and simply output the
objects as if the objects exist.
Overall, this makes sense, but curious to hear what Taylor has to say.
I also might
post a patch series in this direction to consolidate our thoughts and
get a feedback
from the list.
^ permalink raw reply
* Re: [PATCH v3] bulk-checkin: only support blobs in index_bulk_checkin
From: Junio C Hamano @ 2023-09-27 16:26 UTC (permalink / raw)
To: Taylor Blau; +Cc: Eric W. Biederman, brian m. carlson, Elijah Newren, git
In-Reply-To: <ZRQ9aSeu/wpJERuV@nand.local>
Taylor Blau <me@ttaylorr.com> writes:
> I am not sure that I follow. If we have an address in memory from which
> we want to stream raw bytes directly to the packfile, that should work
> for all objects regardless of type, no?
For a single hash world, yes. For keeping track of "the other hash"
and correspondence, you need to (1) interpret the contents of the
object (e.g., if you received a tree contents for SHA-1 repository,
you'd need to split them into tree entries and know which parts of
the bytestream are SHA-1 hashes of the tree contebnts), (2) come
up with the corresponding tree contents in the SHA-256 world (you
should be able to do that now you know SHA-1 names of the objects
directly referred to by the tree) and hash that using SHA-256, and
(3) remember the SHA-1 and the SHA-256 name correspondence of the
tree object you just hashed, in addition to the usual (4) hashing
the contents using SHA-1 hash algorithm without caring what the byte
stream represents.
^ permalink raw reply
* Re: What's cooking in git.git (Sep 2023, #08; Mon, 25)
From: Christian Couder @ 2023-09-27 14:38 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <xmqqpm25pxkh.fsf@gitster.g>
On Tue, Sep 26, 2023 at 4:25 AM Junio C Hamano <gitster@pobox.com> wrote:
> * ks/ref-filter-mailmap (2023-09-25) 3 commits
> - ref-filter: add mailmap support
> - t/t6300: introduce test_bad_atom
> - t/t6300: cleanup test_atom
>
> "git for-each-ref" and friends learn to apply mailmap to authorname
> and other fields.
>
> Will merge to 'next'.
> source: <20230925175050.3498-1-five231003@gmail.com>
Great!
> * ps/revision-cmdline-stdin-not (2023-09-25) 1 commit
> - revision: make pseudo-opt flags read via stdin behave consistently
>
> "git rev-list --stdin" learned to take non-revisions (like "--not")
> recently from the standard input, but the way such a "--not" was
> handled was quite confusing, which has been rethought. This is
> potentially a change that breaks backward compatibility.
>
> Will merge to 'next'?
> source: <6221acd2796853144f8e84081655fbc79fdc6634.1695646898.git.ps@pks.im>
The patch LGTM too.
> * cc/repack-sift-filtered-objects-to-separate-pack (2023-09-25) 10 commits
> - SQUASH??? t0080 is already taken
Yeah, it's taken by js/doc-unit-tests. I am Ok with using t0081 or
something else if someone has a better suggestion. Not sure if I
should resend a v8 or if you would be Ok with just squashing this
yourself when merging.
> - gc: add `gc.repackFilterTo` config option
> - repack: implement `--filter-to` for storing filtered out objects
> - gc: add `gc.repackFilter` config option
> - repack: add `--filter=<filter-spec>` option
> - pack-bitmap-write: rebuild using new bitmap when remapping
> - repack: refactor finding pack prefix
> - repack: refactor finishing pack-objects command
> - t/helper: add 'find-pack' test-tool
> - pack-objects: allow `--filter` without `--stdout`
>
> "git repack" machinery learns to pay attention to the "--filter="
> option.
>
> Looking better.
> source: <20230925152517.803579-1-christian.couder@gmail.com>
> * cc/git-replay (2023-09-07) 15 commits
> - replay: stop assuming replayed branches do not diverge
> - replay: add --contained to rebase contained branches
> - replay: add --advance or 'cherry-pick' mode
> - replay: disallow revision specific options and pathspecs
> - replay: use standard revision ranges
> - replay: make it a minimal server side command
> - replay: remove HEAD related sanity check
> - replay: remove progress and info output
> - replay: add an important FIXME comment about gpg signing
> - replay: don't simplify history
> - replay: introduce pick_regular_commit()
> - replay: die() instead of failing assert()
> - replay: start using parse_options API
> - replay: introduce new builtin
> - t6429: remove switching aspects of fast-rebase
>
> Waiting for review response.
> cf. <52277471-4ddd-b2e0-62ca-c2a5b59ae418@gmx.de>
> cf. <58daa706-7efb-51dd-9061-202ef650b96a@gmx.de>
> cf. <f0e75d47-c277-9fbb-7bcd-53e4e5686f3c@gmx.de>
As no one replied to Dscho and Derrick seems to not be working on Git
anymore, I think I will just send a new version implementing Dscho's
suggestions.
> May want to wait until tb/repack-existing-packs-cleanup stabilizes.
> source: <20230907092521.733746-1-christian.couder@gmail.com>
Not sure I understand this. Did you mistake the "replay" series with
the "repack --filter" series? In any case
tb/repack-existing-packs-cleanup was merged to master.
^ permalink raw reply
* Re: [PATCH v3] bulk-checkin: only support blobs in index_bulk_checkin
From: Taylor Blau @ 2023-09-27 14:34 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Eric W. Biederman, brian m. carlson, Elijah Newren, git
In-Reply-To: <xmqqil7wmf50.fsf@gitster.g>
On Tue, Sep 26, 2023 at 09:08:59PM -0700, Junio C Hamano wrote:
> Taylor Blau <me@ttaylorr.com> writes:
>
> > Hmm. I wonder if retaining some flexibility in the bulk-checkin
> > mechanism may be worthwhile. We discussed at the Contributor's
> > Summit[^1] today that the bulk-checkin system may be a good fit for
> > packing any blobs/trees created by `merge-tree` or `replay` instead of
> > writing them out as loose objects.
>
> But see the last paragraph of my review comments for the earlier
> round upthread. This particular function implements logic that is
> only applicable to blob objects, and streaming trees, commits, and
> tags will need their own separate helper functions. And when they
> are written, the top-level stream_to_pack() function can be
> reintroduced, which will be a thin dispatcher to the four
> type-specific helpers.
I am not sure that I follow. If we have an address in memory from which
we want to stream raw bytes directly to the packfile, that should work
for all objects regardless of type, no?
Having stream_to_pack() take a non-OBJ_BLOB 'type' argument would be OK
provided that the file descriptor 'fd' contains the raw contents of an
object which matches type 'type'.
IIUC, for callers like in the ORT backend which assemble e.g. the raw
bytes of a tree in its merge-ort.c::write_tree() function like so:
for (i = 0; i < nr; i++) {
struct merged_info *mi = versions->items[offset+i].util;
struct version_info *ri = &mi->result;
strbuf_addf(&buf, "%o %s%c", ri->mode,
versions->items[offset+i].string, '\0');
strbuf_add(&buf, ri->oid.hash, hash_size);
}
we'd want some variant of stream_to_pack() that acts on a 'void *,
size_t' pair rather than an 'int (fd), size_t' pair. Likely its
signature would look something like:
/* write raw bytes to a bulk-checkin pack */
static int write_to_pack(struct bulk_checkin_packfile *state,
git_hash_ctx *ctx, off_t *already_hashed_to,
void *ptr, size_t size, enum object_type type,
unsigned flags);
/* write an object from memory to a bulk-checkin pack */
static int deflate_to_pack_mem(struct bulk_checkin_packfile *state,
struct object_id *result_oid,
void *ptr, size_t size,
enum object_type type, unsigned flags);
, where the above are analogous to `stream_to_pack()` and
`deflate_to_pack()`, respectively. ORT would be taught to conditionally
replace calls like:
write_object_file(buf.buf, buf.len, OBJ_TREE, result_oid);
with:
deflate_to_pack_mem(&state, result_oid, buf.buf, buf.len,
OBJ_TREE, HASH_WRITE_OBJECT);
I guess after writing all of that out, you'd never have any callers of
the existing `deflate_to_pack()` function that pass a file descriptor
containing the contents of a non-blob object. So in that sense, I don't
think that my proposal would change anything about this patch.
But I worry that I am missing something here, so having a sanity check
would be appreciated ;-).
Thanks,
Taylor
^ permalink raw reply
* Re: [PATCH v3 5/6] git-std-lib: introduce git standard library
From: Phillip Wood @ 2023-09-27 14:14 UTC (permalink / raw)
To: Calvin Wan, git; +Cc: nasamuffin, jonathantanmy, linusa, vdye
In-Reply-To: <4cefe4f8-04ee-48fb-aee4-07342b7a062f@gmail.com>
Hi Calvin
On 11/09/2023 14:22, Phillip Wood wrote:
> This is looking good, it would be really nice to see a demonstration of
> building git using git-std-lib (with gettext support) in the next
> iteration.
I've just pushed a couple of patches on top of this series to
https://github.com/phillipwood/git/tree/cw/git-std-lib/v3 that show a
possible way to do that.
Rather than re-using LIB_OBJS when building git-std-lib.a it uses a
separate variable to hold the list of sources. git is linked against
git-std-lib.a so we don't have to worry about having the same source
file in two different libraries. The stub objects are moved into their
own library. This enables git to be built using git-std-lib.a with
make git
which will link git against git-std-lib.a built with gettext and tracing
support. To use git-std-lib.a in other programs compile it and
git-stub-lib.a with
make NO_GETTEXT=YesPlease git-std-lib.a git-stub-lib.a
and link your program against both libraries.
The GIT_STD_LIB define is also removed in favor of more stubbing to
avoid the complications of conditional compilation.
Best Wishes
Phillip
^ permalink raw reply
* Re: [silly] loose, pack, and another thing?
From: Christian Couder @ 2023-09-27 13:46 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <xmqqbkdometi.fsf@gitster.g>
On Wed, Sep 27, 2023 at 11:47 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Just wondering if it would help to have the third kind of object
> representation in the object database, sitting next to loose objects
> and packed objects, say .git/objects/verbatim/<hex-object-name> for
> the contents and .git/objects/verbatim/<hex-object-name>.type that
> records "blob", "tree", "commit", or "tag" (in practice, I would
> expect huge "blob" objects would be the only ones that use this
> mechanism).
Yeah, I think it could help handle large blobs. I guess it would rely
on the underlying filesystem to store the object size.
> The contents will be stored verbatim without compression and without
> any object header (i.e., the usual "<type> <length>\0") and the file
> could be "ln"ed (or "cow"ed if the underlying filesystem allows it)
> to materialize it in the working tree if needed.
>
> "fsck" needs to be told about how to verify them. Create the object
> header in-core and hash that, followed by the contents of that file,
> and make sure the result matches the <hex-object-name> part of the
> filename, or something like that.
What happens when they are transferred? Should the remote unpack them
into the same kind of verbatim object?
^ permalink raw reply
* Re: git-retry tool or git.retry config (built-in implementation)?
From: Bagas Sanjaya @ 2023-09-27 13:29 UTC (permalink / raw)
To: Yaroslav Halchenko; +Cc: Git Mailing List, Isaac To, Junio C Hamano
In-Reply-To: <ZRQjKYBLwNg42xAd@bilena>
On 27/09/2023 19:42, Yaroslav Halchenko wrote:
>
> On Wed, 27 Sep 2023, Bagas Sanjaya wrote:
>
>> On Tue, Sep 26, 2023 at 12:47:51PM -0400, Yaroslav Halchenko wrote:
>>> Dear Git Gurus,
>
>>> In DataLad (https://datalad.org) we are doing lots of automated cloning,
>>> fetching etc as part of our CI etc jobs. Once in a while git operations
>>> fail [see e.g. 1], and beg us to retry but we need to know when to
>>> do so, and not do it upon every failed git invocation since some
>>> failures could be legit (repository is gone). While looking how others
>>> solve it we found
>>> https://stackoverflow.com/questions/35014012/git-retry-if-http-request-failed
>>> which pointed to tools like git-retry and later part of
>>> https://chromium.googlesource.com/infra/infra/+/HEAD/go/src/infra/tools/git/retry_regexp.go
>>> which serve as a collection of regexes to be on lookout for to retry.
>
>>> Would that be the "best" strategy currently?
>
>> Looking at the actual git_retry.py script [1], it really just wraps
>> actual Git commands. IMO, git-retry(1) shell script as you mentioned
>> only calls the python version, which adds another level of indirection
>> (why not doing it in pure shell?).
>
> My guess would be that it is just easier to code in Python usually for
> such cases with a "registry" of hits etc. But why not just to strip .py
> from python script which has shebang already and not require bash
> wrapper at all? ;)
>
>> AFAIK, to solve the retrying problem, we need to have a way to tell
>> transport backend (curl/ssh) to resume transfer from the faulty point.
>
> some times it seems not even getting connected (https) entirely and that
> (?) leading to error in the caller above, e.g. from
> https://github.com/datalad/datalad/issues/7485#issuecomment-1735619755
>
> error: Failed to connect to datasets-tests.datalad.org port 443 after 8291 ms: Couldn't connect to server (curl_result = 28, http_code = 0, sha1 = 3980af8de56946a10ff5c48879e5d6025965d936)\nerror: Unable to find 3980af8de56946a10ff5c48879e5d6025965d936 under ...
>
Then the question is how can the server cache generated on-the-fly bundles
in hope that future clients can resume the transfer.
>
>>> As regex matching might eventually break whenever `git` changes
>>> anything in the output messages, I wondered if there could be a more
>>> robust internal implementation in git itself? Similarly git-annex has
>>> annex.retry config setting which sets the count of retries for
>>> "retriable" operations.
>
>> Do you use porcelain interfaces instead of plumbing ones?
>
> I would say -- a "mix".
>
Junio, what do you think of above ideas from Yaroslav?
--
An old man doll... just what I always wanted! - Clara
^ permalink raw reply
* Re: [PATCH] setup: Only allow extenions.objectFormat to be specified once
From: Eric W. Biederman @ 2023-09-27 13:11 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, brian m. carlson
In-Reply-To: <xmqqr0mkmx9b.fsf@gitster.g>
Junio C Hamano <gitster@pobox.com> writes:
> "Eric W. Biederman" <ebiederm@gmail.com> writes:
>
>> Today there is no sanity checking of what happens when
>> extensions.objectFormat is specified multiple times. Catch confused git
>> configurations by only allowing this option to be specified once.
>
> Hmph. I am not sure if this is worth doing, and especially only for
> "objectformat". Do we intend to apply different rules other than
> "you can give it only once" to other extensions, and if so where
> will these rules be catalogued? I do not see particular harm to let
> them follow the usual "last one wins".
>
> If the patch were about trying to make sure that extensions, which
> are inherentaly per-repository, appear only in $GIT_DIR/config and
> complain if the code gets confused and tried to read them from the
> system or global configuration files, I would understand, and
> strongly support such an effort, ithough.
Unless I misread something, the code only checks for [extensions]
in $GIT_DIR/config.
> The real sanity we want to enforce is that what is reported by
> running "git config extensions.objectformat" must match the object
> format that is used in refs and object database.
Agreed. Allowing git config extensions.objectformat to change the
existing value is allowing the repository to be corrupted.
> Manually futzing
> the configuration file and adding an entry with a contradictory
> value certainly is one way to break that sanity, and this patch may
> catch such a breakage, but once we start worrying about manually
> futzing the configuration file, the check added here would easily
> miss if the futzing is done by replacing instead of adding, so I am
> not sure if this extra code is worth its bits.
>
> But perhaps I am missing something and not seeing why it is worth
> insisting on "last one is the first one" for this particular one.
I somewhat have blinders on. There are 3 configuration options I am
concerned with:
extensions.objectFormat
extensions.compatObjectFormat
core.historicObjectFormat (or whatever name we settle on).
One key concern I heard expressed in earlier reviews is that however we
handle these options we handle them in such a way as to give ourselves
room to rise to challenges in the future.
Whatever we do with parsing we have the following logical
constraints:
For extensions.objectFormat: There can only be a single storage hash.
For extensions.compatObjectFormat: There can be no compatibility hash,
there can be a single compatibility hash, and depending how things go
between now and the next hash function transition we might want multiple
compatibility hashes.
For core.historicObjectFormat: There can be no historic hash function, there
can be a single historic hash function, there can be multiple historic
hash functions.
For the compatibility hash I think it is unlikely we will want to
support more than one compatibility hash in practice but I can imagine
a scenario where we just get into the transition from SHA-1 to SHA-256
and a serious break is discovered that requires switching to FutureHash
ASAP.
For historic object formats like SHA-1 will become post transition there
are references embedded in commit comments, email messages, bug
trackers. All kinds of places that we can not update so there is
fundamentally a need to be able to find which current objects correspond
to the historic names. For a project each hash function transition will
create more such objects.
When I looked I saw two ways within current git to specify a list of
values for a single configuration option.
- Give that option multiple times.
- Parse the option value in such a way as to generate a list.
It is my sense just specifying the compatObjectFormat multiple times to
specify multiple compatibility object formats makes the most sense.
Especially as all is needed today is to only allow a single value.
After I had implemented the only allow once logic for compatObjectFormat
I saw that objectFormat had nothing similar, and knowing it is a bug
for multiple objectFormat wrote a patch to enforce only appear once
for objectFormat as well.
For objectFormat I don't care very much. For compatObjectFormat I truly
care, and for even more for the option that allows finding the current
object from a historic oid (even a truncated one) I care very much.
For me the fundamental question is if we allow multiples compatibility
hashes or historical hashes how do we specify them? Have the option
appear more than once? A comma separated list?
Whatever we decided I want to enforce that doesn't appear in current
configurations so we can support for multiples later.
Eric
>> Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
>> ---
>> setup.c | 8 ++++++++
>> 1 file changed, 8 insertions(+)
>>
>> diff --git a/setup.c b/setup.c
>> index 18927a847b86..ef9f79b8885e 100644
>> --- a/setup.c
>> +++ b/setup.c
>> @@ -580,6 +580,7 @@ static enum extension_result handle_extension(const char *var,
>> if (!strcmp(ext, "noop-v1")) {
>> return EXTENSION_OK;
>> } else if (!strcmp(ext, "objectformat")) {
>> + struct string_list_item *item;
>> int format;
>>
>> if (!value)
>> @@ -588,6 +589,13 @@ static enum extension_result handle_extension(const char *var,
>> if (format == GIT_HASH_UNKNOWN)
>> return error(_("invalid value for '%s': '%s'"),
>> "extensions.objectformat", value);
>> + /* Only support objectFormat being specified once. */
>> + for_each_string_list_item(item, &data->v1_only_extensions) {
>> + if (!strcmp(item->string, "objectformat"))
>> + return error(_("'%s' already specified as '%s'"),
>> + "extensions.objectformat",
>> + hash_algos[data->hash_algo].name);
>> + }
>> data->hash_algo = format;
>> return EXTENSION_OK;
>> }
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox