* [PATCH v3 01/13] reftable/pq: use `size_t` to track iterator index
From: Patrick Steinhardt @ 2024-03-04 10:48 UTC (permalink / raw)
To: git; +Cc: Justin Tobler, James Liu
In-Reply-To: <cover.1709548907.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 862 bytes --]
The reftable priority queue is used by the merged iterator to yield
records from its sub-iterators in the expected order. Each entry has a
record corresponding to such a sub-iterator as well as an index that
indicates which sub-iterator the record belongs to. But while the
sub-iterators are tracked with a `size_t`, we store the index as an
`int` in the entry.
Fix this and use `size_t` consistently.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
reftable/pq.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/reftable/pq.h b/reftable/pq.h
index e85bac9b52..9e25a43a36 100644
--- a/reftable/pq.h
+++ b/reftable/pq.h
@@ -12,7 +12,7 @@ license that can be found in the LICENSE file or at
#include "record.h"
struct pq_entry {
- int index;
+ size_t index;
struct reftable_record rec;
};
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* [PATCH v3 02/13] reftable/merged: make `merged_iter` structure private
From: Patrick Steinhardt @ 2024-03-04 10:48 UTC (permalink / raw)
To: git; +Cc: Justin Tobler, James Liu
In-Reply-To: <cover.1709548907.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 1663 bytes --]
The `merged_iter` structure is not used anywhere outside of "merged.c",
but is declared in its header. Move it into the code file so that it is
clear that its implementation details are never exposed to anything.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
reftable/merged.c | 9 +++++++++
reftable/merged.h | 11 +----------
2 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/reftable/merged.c b/reftable/merged.c
index 1aa6cd31b7..12ebd732e8 100644
--- a/reftable/merged.c
+++ b/reftable/merged.c
@@ -17,6 +17,15 @@ license that can be found in the LICENSE file or at
#include "reftable-error.h"
#include "system.h"
+struct merged_iter {
+ struct reftable_iterator *stack;
+ uint32_t hash_id;
+ size_t stack_len;
+ uint8_t typ;
+ int suppress_deletions;
+ struct merged_iter_pqueue pq;
+};
+
static int merged_iter_init(struct merged_iter *mi)
{
for (size_t i = 0; i < mi->stack_len; i++) {
diff --git a/reftable/merged.h b/reftable/merged.h
index 7d9f95d27e..a2571dbc99 100644
--- a/reftable/merged.h
+++ b/reftable/merged.h
@@ -9,7 +9,7 @@ license that can be found in the LICENSE file or at
#ifndef MERGED_H
#define MERGED_H
-#include "pq.h"
+#include "system.h"
struct reftable_merged_table {
struct reftable_table *stack;
@@ -24,15 +24,6 @@ struct reftable_merged_table {
uint64_t max;
};
-struct merged_iter {
- struct reftable_iterator *stack;
- uint32_t hash_id;
- size_t stack_len;
- uint8_t typ;
- int suppress_deletions;
- struct merged_iter_pqueue pq;
-};
-
void merged_table_release(struct reftable_merged_table *mt);
#endif
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* [PATCH v3 03/13] reftable/merged: advance subiter on subsequent iteration
From: Patrick Steinhardt @ 2024-03-04 10:48 UTC (permalink / raw)
To: git; +Cc: Justin Tobler, James Liu
In-Reply-To: <cover.1709548907.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 4307 bytes --]
When advancing the merged iterator, we pop the topmost entry from its
priority queue and then advance the sub-iterator that the entry belongs
to, adding the result as a new entry. This is quite sensible in the case
where the merged iterator is used to actually iterate through records.
But the merged iterator is also used when we look up a single record,
only, so advancing the sub-iterator is wasted effort because we would
never even look at the result.
Instead of immediately advancing the sub-iterator, we can also defer
this to the next iteration of the merged iterator by storing the
intent-to-advance. This results in a small speedup when reading many
records. The following benchmark creates 10000 refs, which will also end
up with many ref lookups:
Benchmark 1: update-ref: create many refs (revision = HEAD~)
Time (mean ± σ): 337.2 ms ± 7.3 ms [User: 200.1 ms, System: 136.9 ms]
Range (min … max): 329.3 ms … 373.2 ms 100 runs
Benchmark 2: update-ref: create many refs (revision = HEAD)
Time (mean ± σ): 332.5 ms ± 5.9 ms [User: 197.2 ms, System: 135.1 ms]
Range (min … max): 327.6 ms … 359.8 ms 100 runs
Summary
update-ref: create many refs (revision = HEAD) ran
1.01 ± 0.03 times faster than update-ref: create many refs (revision = HEAD~)
While this speedup alone isn't really worth it, this refactoring will
also allow two additional optimizations in subsequent patches. First, it
will allow us to special-case when there is only a single sub-iter left
to circumvent the priority queue altogether. And second, it makes it
easier to avoid copying records to the caller.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
reftable/merged.c | 26 ++++++++++++--------------
1 file changed, 12 insertions(+), 14 deletions(-)
diff --git a/reftable/merged.c b/reftable/merged.c
index 12ebd732e8..9b1ccfff00 100644
--- a/reftable/merged.c
+++ b/reftable/merged.c
@@ -19,11 +19,12 @@ license that can be found in the LICENSE file or at
struct merged_iter {
struct reftable_iterator *stack;
+ struct merged_iter_pqueue pq;
uint32_t hash_id;
size_t stack_len;
uint8_t typ;
int suppress_deletions;
- struct merged_iter_pqueue pq;
+ ssize_t advance_index;
};
static int merged_iter_init(struct merged_iter *mi)
@@ -96,13 +97,17 @@ static int merged_iter_next_entry(struct merged_iter *mi,
struct pq_entry entry = { 0 };
int err = 0;
+ if (mi->advance_index >= 0) {
+ err = merged_iter_advance_subiter(mi, mi->advance_index);
+ if (err < 0)
+ return err;
+ mi->advance_index = -1;
+ }
+
if (merged_iter_pqueue_is_empty(mi->pq))
return 1;
entry = merged_iter_pqueue_remove(&mi->pq);
- err = merged_iter_advance_subiter(mi, entry.index);
- if (err < 0)
- return err;
/*
One can also use reftable as datacenter-local storage, where the ref
@@ -116,14 +121,6 @@ static int merged_iter_next_entry(struct merged_iter *mi,
struct pq_entry top = merged_iter_pqueue_top(mi->pq);
int cmp;
- /*
- * When the next entry comes from the same queue as the current
- * entry then it must by definition be larger. This avoids a
- * comparison in the most common case.
- */
- if (top.index == entry.index)
- break;
-
cmp = reftable_record_cmp(&top.rec, &entry.rec);
if (cmp > 0)
break;
@@ -137,6 +134,7 @@ static int merged_iter_next_entry(struct merged_iter *mi,
reftable_record_release(rec);
*rec = entry.rec;
+ mi->advance_index = entry.index;
done:
if (err)
@@ -160,9 +158,8 @@ static int merged_iter_next(struct merged_iter *mi, struct reftable_record *rec)
static int merged_iter_next_void(void *p, struct reftable_record *rec)
{
struct merged_iter *mi = p;
- if (merged_iter_pqueue_is_empty(mi->pq))
+ if (merged_iter_pqueue_is_empty(mi->pq) && mi->advance_index < 0)
return 1;
-
return merged_iter_next(mi, rec);
}
@@ -255,6 +252,7 @@ static int merged_table_seek_record(struct reftable_merged_table *mt,
.typ = reftable_record_type(rec),
.hash_id = mt->hash_id,
.suppress_deletions = mt->suppress_deletions,
+ .advance_index = -1,
};
struct merged_iter *p;
int err;
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* [PATCH v3 04/13] reftable/merged: make subiters own their records
From: Patrick Steinhardt @ 2024-03-04 10:48 UTC (permalink / raw)
To: git; +Cc: Justin Tobler, James Liu
In-Reply-To: <cover.1709548907.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 9248 bytes --]
For each subiterator, the merged table needs to track their current
record. This record is owned by the priority queue though instead of by
the merged iterator. This is not optimal performance-wise.
For one, we need to move around records whenever we add or remove a
record from the priority queue. Thus, the bigger the entries the more
bytes we need to copy around. And compared to pointers, a reftable
record is rather on the bigger side. The other issue is that this makes
it harder to reuse the records.
Refactor the code so that the merged iterator tracks ownership of the
records per-subiter. Instead of having records in the priority queue, we
can now use mere pointers to the per-subiter records. This also allows
us to swap records between the caller and the per-subiter record instead
of doing an actual copy via `reftable_record_copy_from()`, which removes
the need to release the caller-provided record.
This results in a noticeable speedup when iterating through many refs.
The following benchmark iterates through 1 million refs:
Benchmark 1: show-ref: single matching ref (revision = HEAD~)
Time (mean ± σ): 145.5 ms ± 4.5 ms [User: 142.5 ms, System: 2.8 ms]
Range (min … max): 141.3 ms … 177.0 ms 1000 runs
Benchmark 2: show-ref: single matching ref (revision = HEAD)
Time (mean ± σ): 139.0 ms ± 4.7 ms [User: 136.1 ms, System: 2.8 ms]
Range (min … max): 134.2 ms … 182.2 ms 1000 runs
Summary
show-ref: single matching ref (revision = HEAD) ran
1.05 ± 0.05 times faster than show-ref: single matching ref (revision = HEAD~)
This refactoring also allows a subsequent refactoring where we start
reusing memory allocated by the reftable records because we do not need
to release the caller-provided record anymore.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
reftable/merged.c | 54 ++++++++++++++++++++++++----------------------
reftable/pq.c | 8 ++-----
reftable/pq.h | 2 +-
reftable/pq_test.c | 41 ++++++++++++++++-------------------
4 files changed, 49 insertions(+), 56 deletions(-)
diff --git a/reftable/merged.c b/reftable/merged.c
index 9b1ccfff00..ae74234472 100644
--- a/reftable/merged.c
+++ b/reftable/merged.c
@@ -17,8 +17,13 @@ license that can be found in the LICENSE file or at
#include "reftable-error.h"
#include "system.h"
+struct merged_subiter {
+ struct reftable_iterator iter;
+ struct reftable_record rec;
+};
+
struct merged_iter {
- struct reftable_iterator *stack;
+ struct merged_subiter *subiters;
struct merged_iter_pqueue pq;
uint32_t hash_id;
size_t stack_len;
@@ -32,16 +37,18 @@ static int merged_iter_init(struct merged_iter *mi)
for (size_t i = 0; i < mi->stack_len; i++) {
struct pq_entry e = {
.index = i,
+ .rec = &mi->subiters[i].rec,
};
int err;
- reftable_record_init(&e.rec, mi->typ);
- err = iterator_next(&mi->stack[i], &e.rec);
+ reftable_record_init(&mi->subiters[i].rec, mi->typ);
+ err = iterator_next(&mi->subiters[i].iter,
+ &mi->subiters[i].rec);
if (err < 0)
return err;
if (err > 0) {
- reftable_iterator_destroy(&mi->stack[i]);
- reftable_record_release(&e.rec);
+ reftable_iterator_destroy(&mi->subiters[i].iter);
+ reftable_record_release(&mi->subiters[i].rec);
continue;
}
@@ -56,9 +63,11 @@ static void merged_iter_close(void *p)
struct merged_iter *mi = p;
merged_iter_pqueue_release(&mi->pq);
- for (size_t i = 0; i < mi->stack_len; i++)
- reftable_iterator_destroy(&mi->stack[i]);
- reftable_free(mi->stack);
+ for (size_t i = 0; i < mi->stack_len; i++) {
+ reftable_iterator_destroy(&mi->subiters[i].iter);
+ reftable_record_release(&mi->subiters[i].rec);
+ }
+ reftable_free(mi->subiters);
}
static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,
@@ -66,17 +75,16 @@ static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,
{
struct pq_entry e = {
.index = idx,
+ .rec = &mi->subiters[idx].rec,
};
int err;
- reftable_record_init(&e.rec, mi->typ);
- err = iterator_next(&mi->stack[idx], &e.rec);
+ err = iterator_next(&mi->subiters[idx].iter, &mi->subiters[idx].rec);
if (err < 0)
return err;
-
if (err > 0) {
- reftable_iterator_destroy(&mi->stack[idx]);
- reftable_record_release(&e.rec);
+ reftable_iterator_destroy(&mi->subiters[idx].iter);
+ reftable_record_release(&mi->subiters[idx].rec);
return 0;
}
@@ -86,7 +94,7 @@ static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,
static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)
{
- if (iterator_is_null(&mi->stack[idx]))
+ if (iterator_is_null(&mi->subiters[idx].iter))
return 0;
return merged_iter_advance_nonnull_subiter(mi, idx);
}
@@ -121,25 +129,19 @@ static int merged_iter_next_entry(struct merged_iter *mi,
struct pq_entry top = merged_iter_pqueue_top(mi->pq);
int cmp;
- cmp = reftable_record_cmp(&top.rec, &entry.rec);
+ cmp = reftable_record_cmp(top.rec, entry.rec);
if (cmp > 0)
break;
merged_iter_pqueue_remove(&mi->pq);
err = merged_iter_advance_subiter(mi, top.index);
if (err < 0)
- goto done;
- reftable_record_release(&top.rec);
+ return err;
}
- reftable_record_release(rec);
- *rec = entry.rec;
mi->advance_index = entry.index;
-
-done:
- if (err)
- reftable_record_release(&entry.rec);
- return err;
+ SWAP(*rec, *entry.rec);
+ return 0;
}
static int merged_iter_next(struct merged_iter *mi, struct reftable_record *rec)
@@ -257,10 +259,10 @@ static int merged_table_seek_record(struct reftable_merged_table *mt,
struct merged_iter *p;
int err;
- REFTABLE_CALLOC_ARRAY(merged.stack, mt->stack_len);
+ REFTABLE_CALLOC_ARRAY(merged.subiters, mt->stack_len);
for (size_t i = 0; i < mt->stack_len; i++) {
err = reftable_table_seek_record(&mt->stack[i],
- &merged.stack[merged.stack_len], rec);
+ &merged.subiters[merged.stack_len].iter, rec);
if (err < 0)
goto out;
if (!err)
diff --git a/reftable/pq.c b/reftable/pq.c
index e0ccce2b97..0074d6bc43 100644
--- a/reftable/pq.c
+++ b/reftable/pq.c
@@ -14,7 +14,7 @@ license that can be found in the LICENSE file or at
int pq_less(struct pq_entry *a, struct pq_entry *b)
{
- int cmp = reftable_record_cmp(&a->rec, &b->rec);
+ int cmp = reftable_record_cmp(a->rec, b->rec);
if (cmp == 0)
return a->index > b->index;
return cmp < 0;
@@ -82,10 +82,6 @@ void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry
void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)
{
- int i = 0;
- for (i = 0; i < pq->len; i++) {
- reftable_record_release(&pq->heap[i].rec);
- }
FREE_AND_NULL(pq->heap);
- pq->len = pq->cap = 0;
+ memset(pq, 0, sizeof(*pq));
}
diff --git a/reftable/pq.h b/reftable/pq.h
index 9e25a43a36..ce23972c16 100644
--- a/reftable/pq.h
+++ b/reftable/pq.h
@@ -13,7 +13,7 @@ license that can be found in the LICENSE file or at
struct pq_entry {
size_t index;
- struct reftable_record rec;
+ struct reftable_record *rec;
};
struct merged_iter_pqueue {
diff --git a/reftable/pq_test.c b/reftable/pq_test.c
index c202eff848..b7d3c80cc7 100644
--- a/reftable/pq_test.c
+++ b/reftable/pq_test.c
@@ -27,48 +27,43 @@ void merged_iter_pqueue_check(struct merged_iter_pqueue pq)
static void test_pq(void)
{
- char *names[54] = { NULL };
- int N = ARRAY_SIZE(names) - 1;
-
struct merged_iter_pqueue pq = { NULL };
+ struct reftable_record recs[54];
+ int N = ARRAY_SIZE(recs) - 1, i;
char *last = NULL;
- int i = 0;
for (i = 0; i < N; i++) {
- char name[100];
- snprintf(name, sizeof(name), "%02d", i);
- names[i] = xstrdup(name);
+ struct strbuf refname = STRBUF_INIT;
+ strbuf_addf(&refname, "%02d", i);
+
+ reftable_record_init(&recs[i], BLOCK_TYPE_REF);
+ recs[i].u.ref.refname = strbuf_detach(&refname, NULL);
}
i = 1;
do {
- struct pq_entry e = { .rec = { .type = BLOCK_TYPE_REF,
- .u.ref = {
- .refname = names[i],
- } } };
+ struct pq_entry e = {
+ .rec = &recs[i],
+ };
+
merged_iter_pqueue_add(&pq, &e);
merged_iter_pqueue_check(pq);
+
i = (i * 7) % N;
} while (i != 1);
while (!merged_iter_pqueue_is_empty(pq)) {
struct pq_entry e = merged_iter_pqueue_remove(&pq);
- struct reftable_record *rec = &e.rec;
merged_iter_pqueue_check(pq);
- EXPECT(reftable_record_type(rec) == BLOCK_TYPE_REF);
- if (last) {
- EXPECT(strcmp(last, rec->u.ref.refname) < 0);
- }
- /* this is names[i], so don't dealloc. */
- last = rec->u.ref.refname;
- rec->u.ref.refname = NULL;
- reftable_record_release(rec);
- }
- for (i = 0; i < N; i++) {
- reftable_free(names[i]);
+ EXPECT(reftable_record_type(e.rec) == BLOCK_TYPE_REF);
+ if (last)
+ EXPECT(strcmp(last, e.rec->u.ref.refname) < 0);
+ last = e.rec->u.ref.refname;
}
+ for (i = 0; i < N; i++)
+ reftable_record_release(&recs[i]);
merged_iter_pqueue_release(&pq);
}
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* [PATCH v3 05/13] reftable/merged: remove unnecessary null check for subiters
From: Patrick Steinhardt @ 2024-03-04 10:49 UTC (permalink / raw)
To: git; +Cc: Justin Tobler, James Liu
In-Reply-To: <cover.1709548907.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 3069 bytes --]
Whenever we advance a subiter we first call `iterator_is_null()`. This
is not needed though because we only ever advance subiters which have
entries in the priority queue, and we do not end entries to the priority
queue when the subiter has been exhausted.
Drop the check as well as the now-unused function. This results in a
surprisingly big speedup:
Benchmark 1: show-ref: single matching ref (revision = HEAD~)
Time (mean ± σ): 138.1 ms ± 4.4 ms [User: 135.1 ms, System: 2.8 ms]
Range (min … max): 133.4 ms … 167.3 ms 1000 runs
Benchmark 2: show-ref: single matching ref (revision = HEAD)
Time (mean ± σ): 134.4 ms ± 4.2 ms [User: 131.5 ms, System: 2.8 ms]
Range (min … max): 130.0 ms … 164.0 ms 1000 runs
Summary
show-ref: single matching ref (revision = HEAD) ran
1.03 ± 0.05 times faster than show-ref: single matching ref (revision = HEAD~)
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
reftable/iter.c | 5 -----
reftable/iter.h | 4 ----
reftable/merged.c | 10 +---------
3 files changed, 1 insertion(+), 18 deletions(-)
diff --git a/reftable/iter.c b/reftable/iter.c
index 8b5ebf6183..7aa30c4a51 100644
--- a/reftable/iter.c
+++ b/reftable/iter.c
@@ -16,11 +16,6 @@ license that can be found in the LICENSE file or at
#include "reader.h"
#include "reftable-error.h"
-int iterator_is_null(struct reftable_iterator *it)
-{
- return !it->ops;
-}
-
static void filtering_ref_iterator_close(void *iter_arg)
{
struct filtering_ref_iterator *fri = iter_arg;
diff --git a/reftable/iter.h b/reftable/iter.h
index 47d67d84df..537431baba 100644
--- a/reftable/iter.h
+++ b/reftable/iter.h
@@ -16,10 +16,6 @@ license that can be found in the LICENSE file or at
#include "reftable-iterator.h"
#include "reftable-generic.h"
-/* Returns true for a zeroed out iterator, such as the one returned from
- * iterator_destroy. */
-int iterator_is_null(struct reftable_iterator *it);
-
/* iterator that produces only ref records that point to `oid` */
struct filtering_ref_iterator {
int double_check;
diff --git a/reftable/merged.c b/reftable/merged.c
index ae74234472..29ad09f3d8 100644
--- a/reftable/merged.c
+++ b/reftable/merged.c
@@ -70,8 +70,7 @@ static void merged_iter_close(void *p)
reftable_free(mi->subiters);
}
-static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,
- size_t idx)
+static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)
{
struct pq_entry e = {
.index = idx,
@@ -92,13 +91,6 @@ static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,
return 0;
}
-static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)
-{
- if (iterator_is_null(&mi->subiters[idx].iter))
- return 0;
- return merged_iter_advance_nonnull_subiter(mi, idx);
-}
-
static int merged_iter_next_entry(struct merged_iter *mi,
struct reftable_record *rec)
{
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* [PATCH v3 06/13] reftable/merged: handle subiter cleanup on close only
From: Patrick Steinhardt @ 2024-03-04 10:49 UTC (permalink / raw)
To: git; +Cc: Justin Tobler, James Liu
In-Reply-To: <cover.1709548907.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 1470 bytes --]
When advancing one of the subiters fails we immediately release
resources associated with that subiter. This is not necessary though as
we will release these resources when closing the merged iterator anyway.
Drop the logic and only release resources when the merged iterator is
done. This is a mere cleanup that should help reduce the cognitive load
when reading through the code.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
reftable/merged.c | 12 ++----------
1 file changed, 2 insertions(+), 10 deletions(-)
diff --git a/reftable/merged.c b/reftable/merged.c
index 29ad09f3d8..d9ed4a19dd 100644
--- a/reftable/merged.c
+++ b/reftable/merged.c
@@ -46,11 +46,8 @@ static int merged_iter_init(struct merged_iter *mi)
&mi->subiters[i].rec);
if (err < 0)
return err;
- if (err > 0) {
- reftable_iterator_destroy(&mi->subiters[i].iter);
- reftable_record_release(&mi->subiters[i].rec);
+ if (err > 0)
continue;
- }
merged_iter_pqueue_add(&mi->pq, &e);
}
@@ -79,13 +76,8 @@ static int merged_iter_advance_subiter(struct merged_iter *mi, size_t idx)
int err;
err = iterator_next(&mi->subiters[idx].iter, &mi->subiters[idx].rec);
- if (err < 0)
+ if (err)
return err;
- if (err > 0) {
- reftable_iterator_destroy(&mi->subiters[idx].iter);
- reftable_record_release(&mi->subiters[idx].rec);
- return 0;
- }
merged_iter_pqueue_add(&mi->pq, &e);
return 0;
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* [PATCH v3 07/13] reftable/merged: circumvent pqueue with single subiter
From: Patrick Steinhardt @ 2024-03-04 10:49 UTC (permalink / raw)
To: git; +Cc: Justin Tobler, James Liu
In-Reply-To: <cover.1709548907.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 3218 bytes --]
The merged iterator uses a priority queue to order records so that we
can yielid them in the expected order. This priority queue of course
comes with some overhead as we need to add, compare and remove entries
in that priority queue.
In the general case, that overhead cannot really be avoided. But when we
have a single subiter left then there is no need to use the priority
queue anymore because the order is exactly the same as what that subiter
would return.
While having a single subiter may sound like an edge case, it happens
more frequently than one might think. In the most common scenario, you
can expect a repository to have a single large table that contains most
of the records and then a set of smaller tables which contain later
additions to the reftable stack. In this case it is quite likely that we
exhaust subiters of those smaller stacks before exhausting the large
table.
Special-case this and return records directly from the remaining
subiter. This results in a sizeable speedup when iterating over 1m refs
in a repository with a single table:
Benchmark 1: show-ref: single matching ref (revision = HEAD~)
Time (mean ± σ): 135.4 ms ± 4.4 ms [User: 132.5 ms, System: 2.8 ms]
Range (min … max): 131.0 ms … 166.3 ms 1000 runs
Benchmark 2: show-ref: single matching ref (revision = HEAD)
Time (mean ± σ): 126.3 ms ± 3.9 ms [User: 123.3 ms, System: 2.8 ms]
Range (min … max): 122.7 ms … 157.0 ms 1000 runs
Summary
show-ref: single matching ref (revision = HEAD) ran
1.07 ± 0.05 times faster than show-ref: single matching ref (revision = HEAD~)
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
reftable/merged.c | 24 ++++++++++++++++++++++--
1 file changed, 22 insertions(+), 2 deletions(-)
diff --git a/reftable/merged.c b/reftable/merged.c
index d9ed4a19dd..29161a32cf 100644
--- a/reftable/merged.c
+++ b/reftable/merged.c
@@ -87,16 +87,36 @@ static int merged_iter_next_entry(struct merged_iter *mi,
struct reftable_record *rec)
{
struct pq_entry entry = { 0 };
- int err = 0;
+ int err = 0, empty;
+
+ empty = merged_iter_pqueue_is_empty(mi->pq);
if (mi->advance_index >= 0) {
+ /*
+ * When there are no pqueue entries then we only have a single
+ * subiter left. There is no need to use the pqueue in that
+ * case anymore as we know that the subiter will return entries
+ * in the correct order already.
+ *
+ * While this may sound like a very specific edge case, it may
+ * happen more frequently than you think. Most repositories
+ * will end up having a single large base table that contains
+ * most of the refs. It's thus likely that we exhaust all
+ * subiters but the one from that base ref.
+ */
+ if (empty)
+ return iterator_next(&mi->subiters[mi->advance_index].iter,
+ rec);
+
err = merged_iter_advance_subiter(mi, mi->advance_index);
if (err < 0)
return err;
+ if (!err)
+ empty = 0;
mi->advance_index = -1;
}
- if (merged_iter_pqueue_is_empty(mi->pq))
+ if (empty)
return 1;
entry = merged_iter_pqueue_remove(&mi->pq);
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* [PATCH v3 08/13] reftable/merged: avoid duplicate pqueue emptiness check
From: Patrick Steinhardt @ 2024-03-04 10:49 UTC (permalink / raw)
To: git; +Cc: Justin Tobler, James Liu
In-Reply-To: <cover.1709548907.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 2587 bytes --]
When calling `merged_iter_next_void()` we first check whether the iter
has been exhausted already. We already perform this check two levels
down the stack in `merged_iter_next_entry()` though, which makes this
check redundant.
Now if this check was there to accelerate the common case it might have
made sense to keep it. But the iterator being exhausted is rather the
uncommon case because you can expect most reftable stacks to contain
more than two refs.
Simplify the code by removing the check. As `merged_iter_next_void()` is
basically empty except for calling `merged_iter_next()` now, merge these
two functions. This also results in a tiny speedup when iterating over
many refs:
Benchmark 1: show-ref: single matching ref (revision = HEAD~)
Time (mean ± σ): 125.6 ms ± 3.8 ms [User: 122.7 ms, System: 2.8 ms]
Range (min … max): 122.4 ms … 153.4 ms 1000 runs
Benchmark 2: show-ref: single matching ref (revision = HEAD)
Time (mean ± σ): 124.0 ms ± 3.9 ms [User: 121.1 ms, System: 2.8 ms]
Range (min … max): 120.1 ms … 156.4 ms 1000 runs
Summary
show-ref: single matching ref (revision = HEAD) ran
1.01 ± 0.04 times faster than show-ref: single matching ref (revision = HEAD~)
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
reftable/merged.c | 20 ++++++--------------
1 file changed, 6 insertions(+), 14 deletions(-)
diff --git a/reftable/merged.c b/reftable/merged.c
index 29161a32cf..f85a24c678 100644
--- a/reftable/merged.c
+++ b/reftable/merged.c
@@ -148,27 +148,19 @@ static int merged_iter_next_entry(struct merged_iter *mi,
return 0;
}
-static int merged_iter_next(struct merged_iter *mi, struct reftable_record *rec)
+static int merged_iter_next_void(void *p, struct reftable_record *rec)
{
+ struct merged_iter *mi = p;
while (1) {
int err = merged_iter_next_entry(mi, rec);
- if (err == 0 && mi->suppress_deletions &&
- reftable_record_is_deletion(rec)) {
+ if (err)
+ return err;
+ if (mi->suppress_deletions && reftable_record_is_deletion(rec))
continue;
- }
-
- return err;
+ return 0;
}
}
-static int merged_iter_next_void(void *p, struct reftable_record *rec)
-{
- struct merged_iter *mi = p;
- if (merged_iter_pqueue_is_empty(mi->pq) && mi->advance_index < 0)
- return 1;
- return merged_iter_next(mi, rec);
-}
-
static struct reftable_iterator_vtable merged_iter_vtable = {
.next = &merged_iter_next_void,
.close = &merged_iter_close,
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* [PATCH v3 09/13] reftable/record: reuse refname when decoding
From: Patrick Steinhardt @ 2024-03-04 10:49 UTC (permalink / raw)
To: git; +Cc: Justin Tobler, James Liu
In-Reply-To: <cover.1709548907.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 3427 bytes --]
When decoding a reftable record we will first release the user-provided
record and then decode the new record into it. This is quite inefficient
as we basically need to reallocate at least the refname every time.
Refactor the function to start tracking the refname capacity. Like this,
we can stow away the refname, release, restore and then grow the refname
to the required number of bytes via `REFTABLE_ALLOC_GROW()`.
This refactoring is safe to do because all functions that assigning to
the refname will first call `reftable_ref_record_release()`, which will
zero out the complete record after releasing memory.
This change results in a nice speedup when iterating over 1 million
refs:
Benchmark 1: show-ref: single matching ref (revision = HEAD~)
Time (mean ± σ): 124.0 ms ± 3.9 ms [User: 121.1 ms, System: 2.7 ms]
Range (min … max): 120.4 ms … 152.7 ms 1000 runs
Benchmark 2: show-ref: single matching ref (revision = HEAD)
Time (mean ± σ): 114.4 ms ± 3.7 ms [User: 111.5 ms, System: 2.7 ms]
Range (min … max): 111.0 ms … 152.1 ms 1000 runs
Summary
show-ref: single matching ref (revision = HEAD) ran
1.08 ± 0.05 times faster than show-ref: single matching ref (revision = HEAD~)
Furthermore, with this change we now perform a mostly constant number of
allocations when iterating. Before this change:
HEAP SUMMARY:
in use at exit: 13,603 bytes in 125 blocks
total heap usage: 1,006,620 allocs, 1,006,495 frees, 25,398,363 bytes allocated
After this change:
HEAP SUMMARY:
in use at exit: 13,603 bytes in 125 blocks
total heap usage: 6,623 allocs, 6,498 frees, 509,592 bytes allocated
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
reftable/record.c | 16 ++++++++++++----
reftable/reftable-record.h | 1 +
2 files changed, 13 insertions(+), 4 deletions(-)
diff --git a/reftable/record.c b/reftable/record.c
index d6bb42e887..2b52e47c30 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -368,16 +368,24 @@ static int reftable_ref_record_decode(void *rec, struct strbuf key,
struct reftable_ref_record *r = rec;
struct string_view start = in;
uint64_t update_index = 0;
- int n = get_var_int(&update_index, &in);
+ const char *refname = NULL;
+ size_t refname_cap = 0;
+ int n;
+
+ assert(hash_size > 0);
+
+ n = get_var_int(&update_index, &in);
if (n < 0)
return n;
string_view_consume(&in, n);
+ SWAP(refname, r->refname);
+ SWAP(refname_cap, r->refname_cap);
reftable_ref_record_release(r);
+ SWAP(r->refname, refname);
+ SWAP(r->refname_cap, refname_cap);
- assert(hash_size > 0);
-
- r->refname = reftable_malloc(key.len + 1);
+ REFTABLE_ALLOC_GROW(r->refname, key.len + 1, r->refname_cap);
memcpy(r->refname, key.buf, key.len);
r->refname[key.len] = 0;
diff --git a/reftable/reftable-record.h b/reftable/reftable-record.h
index bb6e99acd3..e657001d42 100644
--- a/reftable/reftable-record.h
+++ b/reftable/reftable-record.h
@@ -22,6 +22,7 @@ license that can be found in the LICENSE file or at
/* reftable_ref_record holds a ref database entry target_value */
struct reftable_ref_record {
char *refname; /* Name of the ref, malloced. */
+ size_t refname_cap;
uint64_t update_index; /* Logical timestamp at which this value is
* written */
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* [PATCH v3 10/13] reftable/record: reuse refname when copying
From: Patrick Steinhardt @ 2024-03-04 10:49 UTC (permalink / raw)
To: git; +Cc: Justin Tobler, James Liu
In-Reply-To: <cover.1709548907.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 2044 bytes --]
Do the same optimization as in the preceding commit, but this time for
`reftable_record_copy()`. While not as noticeable, it still results in a
small speedup when iterating over 1 million refs:
Benchmark 1: show-ref: single matching ref (revision = HEAD~)
Time (mean ± σ): 114.0 ms ± 3.8 ms [User: 111.1 ms, System: 2.7 ms]
Range (min … max): 110.9 ms … 144.3 ms 1000 runs
Benchmark 2: show-ref: single matching ref (revision = HEAD)
Time (mean ± σ): 112.5 ms ± 3.7 ms [User: 109.5 ms, System: 2.8 ms]
Range (min … max): 109.2 ms … 140.7 ms 1000 runs
Summary
show-ref: single matching ref (revision = HEAD) ran
1.01 ± 0.05 times faster than show-ref: single matching ref (revision = HEAD~)
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
reftable/record.c | 18 +++++++++++++++---
1 file changed, 15 insertions(+), 3 deletions(-)
diff --git a/reftable/record.c b/reftable/record.c
index 2b52e47c30..12a44f70e5 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -205,14 +205,26 @@ static void reftable_ref_record_copy_from(void *rec, const void *src_rec,
{
struct reftable_ref_record *ref = rec;
const struct reftable_ref_record *src = src_rec;
+ char *refname = NULL;
+ size_t refname_cap = 0;
+
assert(hash_size > 0);
- /* This is simple and correct, but we could probably reuse the hash
- * fields. */
+ SWAP(refname, ref->refname);
+ SWAP(refname_cap, ref->refname_cap);
reftable_ref_record_release(ref);
+ SWAP(ref->refname, refname);
+ SWAP(ref->refname_cap, refname_cap);
+
if (src->refname) {
- ref->refname = xstrdup(src->refname);
+ size_t refname_len = strlen(src->refname);
+
+ REFTABLE_ALLOC_GROW(ref->refname, refname_len + 1,
+ ref->refname_cap);
+ memcpy(ref->refname, src->refname, refname_len);
+ ref->refname[refname_len] = 0;
}
+
ref->update_index = src->update_index;
ref->value_type = src->value_type;
switch (src->value_type) {
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* [PATCH v3 11/13] reftable/record: decode keys in place
From: Patrick Steinhardt @ 2024-03-04 10:49 UTC (permalink / raw)
To: git; +Cc: Justin Tobler, James Liu
In-Reply-To: <cover.1709548907.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 7642 bytes --]
When reading a record from a block, we need to decode the record's key.
As reftable keys are prefix-compressed, meaning they reuse a prefix from
the preceding record's key, this is a bit more involved than just having
to copy the relevant bytes: we need to figure out the prefix and suffix
lengths, copy the prefix from the preceding record and finally copy the
suffix from the current record.
This is done by passing three buffers to `reftable_decode_key()`: one
buffer that holds the result, one buffer that holds the last key, and
one buffer that points to the current record. The final key is then
assembled by calling `strbuf_add()` twice to copy over the prefix and
suffix.
Performing two memory copies is inefficient though. And we can indeed do
better by decoding keys in place. Instead of providing two buffers, the
caller may only call a single buffer that is already pre-populated with
the last key. Like this, we only have to call `strbuf_setlen()` to trim
the record to its prefix and then `strbuf_add()` to add the suffix.
This refactoring leads to a noticeable performance bump when iterating
over 1 million refs:
Benchmark 1: show-ref: single matching ref (revision = HEAD~)
Time (mean ± σ): 112.2 ms ± 3.9 ms [User: 109.3 ms, System: 2.8 ms]
Range (min … max): 109.2 ms … 149.6 ms 1000 runs
Benchmark 2: show-ref: single matching ref (revision = HEAD)
Time (mean ± σ): 106.0 ms ± 3.5 ms [User: 103.2 ms, System: 2.7 ms]
Range (min … max): 103.2 ms … 133.7 ms 1000 runs
Summary
show-ref: single matching ref (revision = HEAD) ran
1.06 ± 0.05 times faster than show-ref: single matching ref (revision = HEAD~)
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
reftable/block.c | 25 +++++++++++--------------
reftable/block.h | 2 --
reftable/record.c | 19 +++++++++----------
reftable/record.h | 9 ++++++---
reftable/record_test.c | 3 ++-
5 files changed, 28 insertions(+), 30 deletions(-)
diff --git a/reftable/block.c b/reftable/block.c
index 72eb73b380..ad9074dba6 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -291,9 +291,8 @@ static int restart_key_less(size_t idx, void *args)
/* the restart key is verbatim in the block, so this could avoid the
alloc for decoding the key */
struct strbuf rkey = STRBUF_INIT;
- struct strbuf last_key = STRBUF_INIT;
uint8_t unused_extra;
- int n = reftable_decode_key(&rkey, &unused_extra, last_key, in);
+ int n = reftable_decode_key(&rkey, &unused_extra, in);
int result;
if (n < 0) {
a->error = 1;
@@ -326,35 +325,34 @@ int block_iter_next(struct block_iter *it, struct reftable_record *rec)
if (it->next_off >= it->br->block_len)
return 1;
- n = reftable_decode_key(&it->key, &extra, it->last_key, in);
+ n = reftable_decode_key(&it->last_key, &extra, in);
if (n < 0)
return -1;
-
- if (!it->key.len)
+ if (!it->last_key.len)
return REFTABLE_FORMAT_ERROR;
string_view_consume(&in, n);
- n = reftable_record_decode(rec, it->key, extra, in, it->br->hash_size);
+ n = reftable_record_decode(rec, it->last_key, extra, in, it->br->hash_size);
if (n < 0)
return -1;
string_view_consume(&in, n);
- strbuf_swap(&it->last_key, &it->key);
it->next_off += start.len - in.len;
return 0;
}
int block_reader_first_key(struct block_reader *br, struct strbuf *key)
{
- struct strbuf empty = STRBUF_INIT;
- int off = br->header_off + 4;
+ int off = br->header_off + 4, n;
struct string_view in = {
.buf = br->block.data + off,
.len = br->block_len - off,
};
-
uint8_t extra = 0;
- int n = reftable_decode_key(key, &extra, empty, in);
+
+ strbuf_reset(key);
+
+ n = reftable_decode_key(key, &extra, in);
if (n < 0)
return n;
if (!key->len)
@@ -371,7 +369,6 @@ int block_iter_seek(struct block_iter *it, struct strbuf *want)
void block_iter_close(struct block_iter *it)
{
strbuf_release(&it->last_key);
- strbuf_release(&it->key);
}
int block_reader_seek(struct block_reader *br, struct block_iter *it,
@@ -408,8 +405,8 @@ int block_reader_seek(struct block_reader *br, struct block_iter *it,
if (err < 0)
goto done;
- reftable_record_key(&rec, &it->key);
- if (err > 0 || strbuf_cmp(&it->key, want) >= 0) {
+ reftable_record_key(&rec, &it->last_key);
+ if (err > 0 || strbuf_cmp(&it->last_key, want) >= 0) {
err = 0;
goto done;
}
diff --git a/reftable/block.h b/reftable/block.h
index 17481e6331..51699af233 100644
--- a/reftable/block.h
+++ b/reftable/block.h
@@ -84,12 +84,10 @@ struct block_iter {
/* key for last entry we read. */
struct strbuf last_key;
- struct strbuf key;
};
#define BLOCK_ITER_INIT { \
.last_key = STRBUF_INIT, \
- .key = STRBUF_INIT, \
}
/* initializes a block reader. */
diff --git a/reftable/record.c b/reftable/record.c
index 12a44f70e5..b9c6eee88a 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -159,20 +159,19 @@ int reftable_encode_key(int *restart, struct string_view dest,
return start.len - dest.len;
}
-int reftable_decode_key(struct strbuf *key, uint8_t *extra,
- struct strbuf last_key, struct string_view in)
+int reftable_decode_key(struct strbuf *last_key, uint8_t *extra,
+ struct string_view in)
{
int start_len = in.len;
uint64_t prefix_len = 0;
uint64_t suffix_len = 0;
- int n = get_var_int(&prefix_len, &in);
+ int n;
+
+ n = get_var_int(&prefix_len, &in);
if (n < 0)
return -1;
string_view_consume(&in, n);
- if (prefix_len > last_key.len)
- return -1;
-
n = get_var_int(&suffix_len, &in);
if (n <= 0)
return -1;
@@ -181,12 +180,12 @@ int reftable_decode_key(struct strbuf *key, uint8_t *extra,
*extra = (uint8_t)(suffix_len & 0x7);
suffix_len >>= 3;
- if (in.len < suffix_len)
+ if (in.len < suffix_len ||
+ prefix_len > last_key->len)
return -1;
- strbuf_reset(key);
- strbuf_add(key, last_key.buf, prefix_len);
- strbuf_add(key, in.buf, suffix_len);
+ strbuf_setlen(last_key, prefix_len);
+ strbuf_add(last_key, in.buf, suffix_len);
string_view_consume(&in, suffix_len);
return start_len - in.len;
diff --git a/reftable/record.h b/reftable/record.h
index a05e2be179..91c9c6ebfd 100644
--- a/reftable/record.h
+++ b/reftable/record.h
@@ -81,9 +81,12 @@ int reftable_encode_key(int *is_restart, struct string_view dest,
struct strbuf prev_key, struct strbuf key,
uint8_t extra);
-/* Decode into `key` and `extra` from `in` */
-int reftable_decode_key(struct strbuf *key, uint8_t *extra,
- struct strbuf last_key, struct string_view in);
+/*
+ * Decode into `last_key` and `extra` from `in`. `last_key` is expected to
+ * contain the decoded key of the preceding record, if any.
+ */
+int reftable_decode_key(struct strbuf *last_key, uint8_t *extra,
+ struct string_view in);
/* reftable_index_record are used internally to speed up lookups. */
struct reftable_index_record {
diff --git a/reftable/record_test.c b/reftable/record_test.c
index a86cff5526..89209894d8 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -295,7 +295,8 @@ static void test_key_roundtrip(void)
EXPECT(!restart);
EXPECT(n > 0);
- m = reftable_decode_key(&roundtrip, &rt_extra, last_key, dest);
+ strbuf_addstr(&roundtrip, "refs/heads/master");
+ m = reftable_decode_key(&roundtrip, &rt_extra, dest);
EXPECT(n == m);
EXPECT(0 == strbuf_cmp(&key, &roundtrip));
EXPECT(rt_extra == extra);
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* [PATCH v3 12/13] reftable: allow inlining of a few functions
From: Patrick Steinhardt @ 2024-03-04 10:49 UTC (permalink / raw)
To: git; +Cc: Justin Tobler, James Liu
In-Reply-To: <cover.1709548907.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 5049 bytes --]
We have a few functions which are basically just accessors to
structures. As those functions are executed inside the hot loop when
iterating through many refs, the fact that they cannot be inlined is
costing us some performance.
Move the function definitions into their respective headers so that they
can be inlined. This results in a performance improvement when iterating
over 1 million refs:
Benchmark 1: show-ref: single matching ref (revision = HEAD~)
Time (mean ± σ): 105.9 ms ± 3.6 ms [User: 103.0 ms, System: 2.8 ms]
Range (min … max): 103.1 ms … 133.4 ms 1000 runs
Benchmark 2: show-ref: single matching ref (revision = HEAD)
Time (mean ± σ): 100.7 ms ± 3.4 ms [User: 97.8 ms, System: 2.8 ms]
Range (min … max): 97.8 ms … 124.0 ms 1000 runs
Summary
show-ref: single matching ref (revision = HEAD) ran
1.05 ± 0.05 times faster than show-ref: single matching ref (revision = HEAD~)
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
reftable/pq.c | 10 ----------
reftable/pq.h | 12 ++++++++++--
reftable/record.c | 11 -----------
reftable/record.h | 12 ++++++++++--
4 files changed, 20 insertions(+), 25 deletions(-)
diff --git a/reftable/pq.c b/reftable/pq.c
index 0074d6bc43..7fb45d8c60 100644
--- a/reftable/pq.c
+++ b/reftable/pq.c
@@ -20,16 +20,6 @@ int pq_less(struct pq_entry *a, struct pq_entry *b)
return cmp < 0;
}
-struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq)
-{
- return pq.heap[0];
-}
-
-int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq)
-{
- return pq.len == 0;
-}
-
struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)
{
int i = 0;
diff --git a/reftable/pq.h b/reftable/pq.h
index ce23972c16..f796c23179 100644
--- a/reftable/pq.h
+++ b/reftable/pq.h
@@ -22,12 +22,20 @@ struct merged_iter_pqueue {
size_t cap;
};
-struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq);
-int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq);
void merged_iter_pqueue_check(struct merged_iter_pqueue pq);
struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq);
void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, const struct pq_entry *e);
void merged_iter_pqueue_release(struct merged_iter_pqueue *pq);
int pq_less(struct pq_entry *a, struct pq_entry *b);
+static inline struct pq_entry merged_iter_pqueue_top(struct merged_iter_pqueue pq)
+{
+ return pq.heap[0];
+}
+
+static inline int merged_iter_pqueue_is_empty(struct merged_iter_pqueue pq)
+{
+ return pq.len == 0;
+}
+
#endif
diff --git a/reftable/record.c b/reftable/record.c
index b9c6eee88a..367de04600 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -1176,11 +1176,6 @@ void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
reftable_record_vtable(rec)->key(reftable_record_data(rec), dest);
}
-uint8_t reftable_record_type(struct reftable_record *rec)
-{
- return rec->type;
-}
-
int reftable_record_encode(struct reftable_record *rec, struct string_view dest,
int hash_size)
{
@@ -1302,12 +1297,6 @@ int reftable_log_record_is_deletion(const struct reftable_log_record *log)
return (log->value_type == REFTABLE_LOG_DELETION);
}
-void string_view_consume(struct string_view *s, int n)
-{
- s->buf += n;
- s->len -= n;
-}
-
static void *reftable_record_data(struct reftable_record *rec)
{
switch (rec->type) {
diff --git a/reftable/record.h b/reftable/record.h
index 91c9c6ebfd..5e8304e052 100644
--- a/reftable/record.h
+++ b/reftable/record.h
@@ -25,7 +25,11 @@ struct string_view {
};
/* Advance `s.buf` by `n`, and decrease length. */
-void string_view_consume(struct string_view *s, int n);
+static inline void string_view_consume(struct string_view *s, int n)
+{
+ s->buf += n;
+ s->len -= n;
+}
/* utilities for de/encoding varints */
@@ -127,7 +131,6 @@ int reftable_record_cmp(struct reftable_record *a, struct reftable_record *b);
int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);
void reftable_record_print(struct reftable_record *rec, int hash_size);
void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
-uint8_t reftable_record_type(struct reftable_record *rec);
void reftable_record_copy_from(struct reftable_record *rec,
struct reftable_record *src, int hash_size);
uint8_t reftable_record_val_type(struct reftable_record *rec);
@@ -138,6 +141,11 @@ int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
int hash_size);
int reftable_record_is_deletion(struct reftable_record *rec);
+static inline uint8_t reftable_record_type(struct reftable_record *rec)
+{
+ return rec->type;
+}
+
/* frees and zeroes out the embedded record */
void reftable_record_release(struct reftable_record *rec);
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* [PATCH v3 13/13] refs/reftable: precompute prefix length
From: Patrick Steinhardt @ 2024-03-04 10:49 UTC (permalink / raw)
To: git; +Cc: Justin Tobler, James Liu
In-Reply-To: <cover.1709548907.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 2001 bytes --]
We're recomputing the prefix length on every iteration of the ref
iterator. Precompute it for another speedup when iterating over 1
million refs:
Benchmark 1: show-ref: single matching ref (revision = HEAD~)
Time (mean ± σ): 100.3 ms ± 3.7 ms [User: 97.3 ms, System: 2.8 ms]
Range (min … max): 97.5 ms … 139.7 ms 1000 runs
Benchmark 2: show-ref: single matching ref (revision = HEAD)
Time (mean ± σ): 95.8 ms ± 3.4 ms [User: 92.9 ms, System: 2.8 ms]
Range (min … max): 93.0 ms … 121.9 ms 1000 runs
Summary
show-ref: single matching ref (revision = HEAD) ran
1.05 ± 0.05 times faster than show-ref: single matching ref (revision = HEAD~)
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
refs/reftable-backend.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index 6c11c4a5e3..249a618b5a 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -346,6 +346,7 @@ struct reftable_ref_iterator {
struct object_id oid;
const char *prefix;
+ size_t prefix_len;
unsigned int flags;
int err;
};
@@ -371,8 +372,8 @@ static int reftable_ref_iterator_advance(struct ref_iterator *ref_iterator)
if (!starts_with(iter->ref.refname, "refs/"))
continue;
- if (iter->prefix &&
- strncmp(iter->prefix, iter->ref.refname, strlen(iter->prefix))) {
+ if (iter->prefix_len &&
+ strncmp(iter->prefix, iter->ref.refname, iter->prefix_len)) {
iter->err = 1;
break;
}
@@ -481,6 +482,7 @@ static struct reftable_ref_iterator *ref_iterator_for_stack(struct reftable_ref_
iter = xcalloc(1, sizeof(*iter));
base_ref_iterator_init(&iter->base, &reftable_ref_iterator_vtable);
iter->prefix = prefix;
+ iter->prefix_len = prefix ? strlen(prefix) : 0;
iter->base.oid = &iter->oid;
iter->flags = flags;
iter->refs = refs;
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* [PATCH 0/4] reftable/stack: register temporary files
From: Patrick Steinhardt @ 2024-03-04 11:10 UTC (permalink / raw)
To: git
[-- Attachment #1: Type: text/plain, Size: 1109 bytes --]
Hi,
when writing to a reftable stack there are several kinds of temporary
files that we have to write. For one this is the new table, second the
new "tables.list.lock" file. We do not register all of these with the
tempfiles subsystem though, which can have the consequence that
interrupted writes may leave those files behind.
This patch series addresses those cases for three cases:
- Newly written tables.
- Newly compacted tables.
- Pending "tables.list.lock" during compaction.
The end result should be more robust regarding interruption and leave
left cruft behind.
Patrick
Patrick Steinhardt (4):
lockfile: report when rollback fails
reftable/stack: register new tables as tempfiles
reftable/stack: register lockfiles during compaction
reftable/stack: register compacted tables as tempfiles
lockfile.h | 4 +-
reftable/stack.c | 329 ++++++++++++++++++++++------------------------
reftable/system.h | 2 +
tempfile.c | 21 +--
tempfile.h | 2 +-
5 files changed, 177 insertions(+), 181 deletions(-)
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* [PATCH 1/4] lockfile: report when rollback fails
From: Patrick Steinhardt @ 2024-03-04 11:10 UTC (permalink / raw)
To: git
In-Reply-To: <cover.1709549619.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 2959 bytes --]
We do not report to the caller when rolling back a lockfile fails, which
will be needed by the reftable compaction logic in a subsequent commit.
It also cannot really report on all errors because the function calls
`delete_tempfile()`, which doesn't return an error either.
Refactor the code so that both `delete_tempfile()` and
`rollback_lock_file()` return an error code.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
lockfile.h | 4 ++--
tempfile.c | 21 +++++++++++++--------
tempfile.h | 2 +-
3 files changed, 16 insertions(+), 11 deletions(-)
diff --git a/lockfile.h b/lockfile.h
index 90af4e66b2..4ed570d3f7 100644
--- a/lockfile.h
+++ b/lockfile.h
@@ -323,9 +323,9 @@ static inline int commit_lock_file_to(struct lock_file *lk, const char *path)
* for a `lock_file` object that has already been committed or rolled
* back.
*/
-static inline void rollback_lock_file(struct lock_file *lk)
+static inline int rollback_lock_file(struct lock_file *lk)
{
- delete_tempfile(&lk->tempfile);
+ return delete_tempfile(&lk->tempfile);
}
#endif /* LOCKFILE_H */
diff --git a/tempfile.c b/tempfile.c
index ecdebf1afb..ed88cf8431 100644
--- a/tempfile.c
+++ b/tempfile.c
@@ -50,15 +50,17 @@
static VOLATILE_LIST_HEAD(tempfile_list);
-static void remove_template_directory(struct tempfile *tempfile,
+static int remove_template_directory(struct tempfile *tempfile,
int in_signal_handler)
{
if (tempfile->directory) {
if (in_signal_handler)
- rmdir(tempfile->directory);
+ return rmdir(tempfile->directory);
else
- rmdir_or_warn(tempfile->directory);
+ return rmdir_or_warn(tempfile->directory);
}
+
+ return 0;
}
static void remove_tempfiles(int in_signal_handler)
@@ -353,16 +355,19 @@ int rename_tempfile(struct tempfile **tempfile_p, const char *path)
return 0;
}
-void delete_tempfile(struct tempfile **tempfile_p)
+int delete_tempfile(struct tempfile **tempfile_p)
{
struct tempfile *tempfile = *tempfile_p;
+ int err = 0;
if (!is_tempfile_active(tempfile))
- return;
+ return 0;
- close_tempfile_gently(tempfile);
- unlink_or_warn(tempfile->filename.buf);
- remove_template_directory(tempfile, 0);
+ err |= close_tempfile_gently(tempfile);
+ err |= unlink_or_warn(tempfile->filename.buf);
+ err |= remove_template_directory(tempfile, 0);
deactivate_tempfile(tempfile);
*tempfile_p = NULL;
+
+ return err ? -1 : 0;
}
diff --git a/tempfile.h b/tempfile.h
index d0413af733..2d2ae5b657 100644
--- a/tempfile.h
+++ b/tempfile.h
@@ -269,7 +269,7 @@ int reopen_tempfile(struct tempfile *tempfile);
* `delete_tempfile()` for a `tempfile` object that has already been
* deleted or renamed.
*/
-void delete_tempfile(struct tempfile **tempfile_p);
+int delete_tempfile(struct tempfile **tempfile_p);
/*
* Close the file descriptor and/or file pointer if they are still
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* [PATCH 2/4] reftable/stack: register new tables as tempfiles
From: Patrick Steinhardt @ 2024-03-04 11:10 UTC (permalink / raw)
To: git
In-Reply-To: <cover.1709549619.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 3253 bytes --]
We do not register new tables which we're about to add to the stack with
the tempfile API. Those tables will thus not be deleted in case Git gets
killed.
Refactor the code to register tables as tempfiles.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
reftable/stack.c | 29 ++++++++++++-----------------
1 file changed, 12 insertions(+), 17 deletions(-)
diff --git a/reftable/stack.c b/reftable/stack.c
index b64e55648a..81544fbfa0 100644
--- a/reftable/stack.c
+++ b/reftable/stack.c
@@ -737,8 +737,9 @@ int reftable_addition_add(struct reftable_addition *add,
struct strbuf tab_file_name = STRBUF_INIT;
struct strbuf next_name = STRBUF_INIT;
struct reftable_writer *wr = NULL;
+ struct tempfile *tab_file = NULL;
int err = 0;
- int tab_fd = 0;
+ int tab_fd;
strbuf_reset(&next_name);
format_name(&next_name, add->next_update_index, add->next_update_index);
@@ -746,17 +747,20 @@ int reftable_addition_add(struct reftable_addition *add,
stack_filename(&temp_tab_file_name, add->stack, next_name.buf);
strbuf_addstr(&temp_tab_file_name, ".temp.XXXXXX");
- tab_fd = mkstemp(temp_tab_file_name.buf);
- if (tab_fd < 0) {
+ tab_file = mks_tempfile(temp_tab_file_name.buf);
+ if (!tab_file) {
err = REFTABLE_IO_ERROR;
goto done;
}
if (add->stack->config.default_permissions) {
- if (chmod(temp_tab_file_name.buf, add->stack->config.default_permissions)) {
+ if (chmod(get_tempfile_path(tab_file),
+ add->stack->config.default_permissions)) {
err = REFTABLE_IO_ERROR;
goto done;
}
}
+ tab_fd = get_tempfile_fd(tab_file);
+
wr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd,
&add->stack->config);
err = write_table(wr, arg);
@@ -771,14 +775,13 @@ int reftable_addition_add(struct reftable_addition *add,
if (err < 0)
goto done;
- err = close(tab_fd);
- tab_fd = 0;
+ err = close_tempfile_gently(tab_file);
if (err < 0) {
err = REFTABLE_IO_ERROR;
goto done;
}
- err = stack_check_addition(add->stack, temp_tab_file_name.buf);
+ err = stack_check_addition(add->stack, get_tempfile_path(tab_file));
if (err < 0)
goto done;
@@ -789,14 +792,13 @@ int reftable_addition_add(struct reftable_addition *add,
format_name(&next_name, wr->min_update_index, wr->max_update_index);
strbuf_addstr(&next_name, ".ref");
-
stack_filename(&tab_file_name, add->stack, next_name.buf);
/*
On windows, this relies on rand() picking a unique destination name.
Maybe we should do retry loop as well?
*/
- err = rename(temp_tab_file_name.buf, tab_file_name.buf);
+ err = rename_tempfile(&tab_file, tab_file_name.buf);
if (err < 0) {
err = REFTABLE_IO_ERROR;
goto done;
@@ -806,14 +808,7 @@ int reftable_addition_add(struct reftable_addition *add,
add->new_tables_cap);
add->new_tables[add->new_tables_len++] = strbuf_detach(&next_name, NULL);
done:
- if (tab_fd > 0) {
- close(tab_fd);
- tab_fd = 0;
- }
- if (temp_tab_file_name.len > 0) {
- unlink(temp_tab_file_name.buf);
- }
-
+ delete_tempfile(&tab_file);
strbuf_release(&temp_tab_file_name);
strbuf_release(&tab_file_name);
strbuf_release(&next_name);
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* [PATCH 3/4] reftable/stack: register lockfiles during compaction
From: Patrick Steinhardt @ 2024-03-04 11:10 UTC (permalink / raw)
To: git
In-Reply-To: <cover.1709549619.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 10974 bytes --]
We do not register any of the locks we acquire when compacting the
reftable stack via our lockfiles interfaces. These locks will thus not
be released when Git gets killed.
Refactor the code to register locks as lockfiles.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
reftable/stack.c | 255 ++++++++++++++++++++++------------------------
reftable/system.h | 2 +
2 files changed, 123 insertions(+), 134 deletions(-)
diff --git a/reftable/stack.c b/reftable/stack.c
index 81544fbfa0..977336b7d5 100644
--- a/reftable/stack.c
+++ b/reftable/stack.c
@@ -978,212 +978,199 @@ static int stack_compact_range(struct reftable_stack *st,
size_t first, size_t last,
struct reftable_log_expiry_config *expiry)
{
- char **delete_on_success = NULL, **subtable_locks = NULL, **listp = NULL;
- struct strbuf temp_tab_file_name = STRBUF_INIT;
+ struct strbuf tables_list_buf = STRBUF_INIT;
+ struct strbuf new_table_temp_path = STRBUF_INIT;
struct strbuf new_table_name = STRBUF_INIT;
- struct strbuf lock_file_name = STRBUF_INIT;
- struct strbuf ref_list_contents = STRBUF_INIT;
struct strbuf new_table_path = STRBUF_INIT;
- size_t i, j, compact_count;
- int err = 0;
- int have_lock = 0;
- int lock_file_fd = -1;
- int is_empty_table = 0;
+ struct strbuf table_name = STRBUF_INIT;
+ struct lock_file tables_list_lock = LOCK_INIT;
+ struct lock_file *table_locks = NULL;
+ int is_empty_table = 0, err = 0;
+ size_t i;
if (first > last || (!expiry && first == last)) {
err = 0;
goto done;
}
- compact_count = last - first + 1;
- REFTABLE_CALLOC_ARRAY(delete_on_success, compact_count + 1);
- REFTABLE_CALLOC_ARRAY(subtable_locks, compact_count + 1);
-
st->stats.attempts++;
- strbuf_reset(&lock_file_name);
- strbuf_addstr(&lock_file_name, st->list_file);
- strbuf_addstr(&lock_file_name, ".lock");
-
- lock_file_fd =
- open(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0666);
- if (lock_file_fd < 0) {
- if (errno == EEXIST) {
+ /*
+ * Hold the lock so that we can read "tables.list" and lock all tables
+ * which are part of the user-specified range.
+ */
+ err = hold_lock_file_for_update(&tables_list_lock, st->list_file,
+ LOCK_NO_DEREF);
+ if (err < 0) {
+ if (errno == EEXIST)
err = 1;
- } else {
+ else
err = REFTABLE_IO_ERROR;
- }
goto done;
}
- /* Don't want to write to the lock for now. */
- close(lock_file_fd);
- lock_file_fd = -1;
- have_lock = 1;
err = stack_uptodate(st);
- if (err != 0)
+ if (err)
goto done;
- for (i = first, j = 0; i <= last; i++) {
- struct strbuf subtab_file_name = STRBUF_INIT;
- struct strbuf subtab_lock = STRBUF_INIT;
- int sublock_file_fd = -1;
-
- stack_filename(&subtab_file_name, st,
- reader_name(st->readers[i]));
-
- strbuf_reset(&subtab_lock);
- strbuf_addbuf(&subtab_lock, &subtab_file_name);
- strbuf_addstr(&subtab_lock, ".lock");
+ /*
+ * Lock all tables in the user-provided range. This is the slice of our
+ * stack which we'll compact.
+ */
+ REFTABLE_CALLOC_ARRAY(table_locks, last - first + 1);
+ for (i = first; i <= last; i++) {
+ stack_filename(&table_name, st, reader_name(st->readers[i]));
- sublock_file_fd = open(subtab_lock.buf,
- O_EXCL | O_CREAT | O_WRONLY, 0666);
- if (sublock_file_fd >= 0) {
- close(sublock_file_fd);
- } else if (sublock_file_fd < 0) {
- if (errno == EEXIST) {
+ err = hold_lock_file_for_update(&table_locks[i - first],
+ table_name.buf, LOCK_NO_DEREF);
+ if (err < 0) {
+ if (errno == EEXIST)
err = 1;
- } else {
+ else
err = REFTABLE_IO_ERROR;
- }
+ goto done;
}
- subtable_locks[j] = subtab_lock.buf;
- delete_on_success[j] = subtab_file_name.buf;
- j++;
-
- if (err != 0)
+ /*
+ * We need to close the lockfiles as we might otherwise easily
+ * run into file descriptor exhaustion when we compress a lot
+ * of tables.
+ */
+ err = close_lock_file_gently(&table_locks[i - first]);
+ if (err < 0) {
+ err = REFTABLE_IO_ERROR;
goto done;
+ }
}
- err = unlink(lock_file_name.buf);
- if (err < 0)
+ /*
+ * We have locked all tables in our range and can thus release the
+ * "tables.list" lock while compacting the locked tables. This allows
+ * concurrent updates to the stack to proceed.
+ */
+ err = rollback_lock_file(&tables_list_lock);
+ if (err < 0) {
+ err = REFTABLE_IO_ERROR;
goto done;
- have_lock = 0;
-
- err = stack_compact_locked(st, first, last, &temp_tab_file_name,
- expiry);
- /* Compaction + tombstones can create an empty table out of non-empty
- * tables. */
- is_empty_table = (err == REFTABLE_EMPTY_TABLE_ERROR);
- if (is_empty_table) {
- err = 0;
}
- if (err < 0)
- goto done;
- lock_file_fd =
- open(lock_file_name.buf, O_EXCL | O_CREAT | O_WRONLY, 0666);
- if (lock_file_fd < 0) {
- if (errno == EEXIST) {
+ /*
+ * Compact the now-locked tables into a new table. Note that compacting
+ * these tables may end up with an empty new table in case tombstones
+ * end up cancelling out all refs in that range.
+ */
+ err = stack_compact_locked(st, first, last, &new_table_temp_path, expiry);
+ if (err < 0) {
+ if (err != REFTABLE_EMPTY_TABLE_ERROR)
+ goto done;
+ is_empty_table = 1;
+ }
+
+ /*
+ * Now that we have written the new, compacted table we need to re-lock
+ * "tables.list". We'll then replace the compacted range of tables with
+ * the new table.
+ */
+ err = hold_lock_file_for_update(&tables_list_lock, st->list_file,
+ LOCK_NO_DEREF);
+ if (err < 0) {
+ if (errno == EEXIST)
err = 1;
- } else {
+ else
err = REFTABLE_IO_ERROR;
- }
goto done;
}
- have_lock = 1;
+
if (st->config.default_permissions) {
- if (chmod(lock_file_name.buf, st->config.default_permissions) < 0) {
+ if (chmod(get_lock_file_path(&tables_list_lock),
+ st->config.default_permissions) < 0) {
err = REFTABLE_IO_ERROR;
goto done;
}
}
- format_name(&new_table_name, st->readers[first]->min_update_index,
- st->readers[last]->max_update_index);
- strbuf_addstr(&new_table_name, ".ref");
-
- stack_filename(&new_table_path, st, new_table_name.buf);
-
+ /*
+ * If the resulting compacted table is not empty, then we need to move
+ * it into place now.
+ */
if (!is_empty_table) {
- /* retry? */
- err = rename(temp_tab_file_name.buf, new_table_path.buf);
+ format_name(&new_table_name, st->readers[first]->min_update_index,
+ st->readers[last]->max_update_index);
+ strbuf_addstr(&new_table_name, ".ref");
+ stack_filename(&new_table_path, st, new_table_name.buf);
+
+ err = rename(new_table_temp_path.buf, new_table_path.buf);
if (err < 0) {
err = REFTABLE_IO_ERROR;
goto done;
}
}
- for (i = 0; i < first; i++) {
- strbuf_addstr(&ref_list_contents, st->readers[i]->name);
- strbuf_addstr(&ref_list_contents, "\n");
- }
- if (!is_empty_table) {
- strbuf_addbuf(&ref_list_contents, &new_table_name);
- strbuf_addstr(&ref_list_contents, "\n");
- }
- for (i = last + 1; i < st->merged->stack_len; i++) {
- strbuf_addstr(&ref_list_contents, st->readers[i]->name);
- strbuf_addstr(&ref_list_contents, "\n");
- }
-
- err = write_in_full(lock_file_fd, ref_list_contents.buf, ref_list_contents.len);
- if (err < 0) {
- err = REFTABLE_IO_ERROR;
- unlink(new_table_path.buf);
- goto done;
- }
-
- err = fsync_component(FSYNC_COMPONENT_REFERENCE, lock_file_fd);
+ /*
+ * Write the new "tables.list" contents with the compacted table we
+ * have just written. In case the compacted table became empty we
+ * simply skip writing it.
+ */
+ for (i = 0; i < first; i++)
+ strbuf_addf(&tables_list_buf, "%s\n", st->readers[i]->name);
+ if (!is_empty_table)
+ strbuf_addf(&tables_list_buf, "%s\n", new_table_name.buf);
+ for (i = last + 1; i < st->merged->stack_len; i++)
+ strbuf_addf(&tables_list_buf, "%s\n", st->readers[i]->name);
+
+ err = write_in_full(get_lock_file_fd(&tables_list_lock),
+ tables_list_buf.buf, tables_list_buf.len);
if (err < 0) {
err = REFTABLE_IO_ERROR;
unlink(new_table_path.buf);
goto done;
}
- err = close(lock_file_fd);
- lock_file_fd = -1;
+ err = fsync_component(FSYNC_COMPONENT_REFERENCE, get_lock_file_fd(&tables_list_lock));
if (err < 0) {
err = REFTABLE_IO_ERROR;
unlink(new_table_path.buf);
goto done;
}
- err = rename(lock_file_name.buf, st->list_file);
+ err = commit_lock_file(&tables_list_lock);
if (err < 0) {
err = REFTABLE_IO_ERROR;
unlink(new_table_path.buf);
goto done;
}
- have_lock = 0;
- /* Reload the stack before deleting. On windows, we can only delete the
- files after we closed them.
- */
+ /*
+ * Reload the stack before deleting the compacted tables. We can only
+ * delete the files after we closed them on Windows, so this needs to
+ * happen first.
+ */
err = reftable_stack_reload_maybe_reuse(st, first < last);
+ if (err < 0)
+ goto done;
- listp = delete_on_success;
- while (*listp) {
- if (strcmp(*listp, new_table_path.buf)) {
- unlink(*listp);
- }
- listp++;
+ /*
+ * Delete the old tables. They may still be in use by concurrent
+ * readers, so it is expected that unlinking tables may fail.
+ */
+ for (i = first; i <= last; i++) {
+ struct lock_file *table_lock = &table_locks[i - first];
+ char *table_path = get_locked_file_path(table_lock);
+ unlink(table_path);
+ free(table_path);
}
done:
- free_names(delete_on_success);
+ rollback_lock_file(&tables_list_lock);
+ for (i = first; table_locks && i <= last; i++)
+ rollback_lock_file(&table_locks[i - first]);
+ reftable_free(table_locks);
- if (subtable_locks) {
- listp = subtable_locks;
- while (*listp) {
- unlink(*listp);
- listp++;
- }
- free_names(subtable_locks);
- }
- if (lock_file_fd >= 0) {
- close(lock_file_fd);
- lock_file_fd = -1;
- }
- if (have_lock) {
- unlink(lock_file_name.buf);
- }
strbuf_release(&new_table_name);
strbuf_release(&new_table_path);
- strbuf_release(&ref_list_contents);
- strbuf_release(&temp_tab_file_name);
- strbuf_release(&lock_file_name);
+ strbuf_release(&new_table_temp_path);
+ strbuf_release(&tables_list_buf);
+ strbuf_release(&table_name);
return err;
}
diff --git a/reftable/system.h b/reftable/system.h
index 6b74a81514..5d8b6dede5 100644
--- a/reftable/system.h
+++ b/reftable/system.h
@@ -12,7 +12,9 @@ license that can be found in the LICENSE file or at
/* This header glues the reftable library to the rest of Git */
#include "git-compat-util.h"
+#include "lockfile.h"
#include "strbuf.h"
+#include "tempfile.h"
#include "hash-ll.h" /* hash ID, sizes.*/
#include "dir.h" /* remove_dir_recursively, for tests.*/
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* [PATCH 4/4] reftable/stack: register compacted tables as tempfiles
From: Patrick Steinhardt @ 2024-03-04 11:10 UTC (permalink / raw)
To: git
In-Reply-To: <cover.1709549619.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 4529 bytes --]
We do not register tables resulting from stack compaction with the
tempfile API. Those tables will thus not be deleted in case Git gets
killed.
Refactor the code to register compacted tables as tempfiles.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
reftable/stack.c | 53 +++++++++++++++++++++++++++---------------------
1 file changed, 30 insertions(+), 23 deletions(-)
diff --git a/reftable/stack.c b/reftable/stack.c
index 977336b7d5..40129da16c 100644
--- a/reftable/stack.c
+++ b/reftable/stack.c
@@ -827,51 +827,57 @@ uint64_t reftable_stack_next_update_index(struct reftable_stack *st)
static int stack_compact_locked(struct reftable_stack *st,
size_t first, size_t last,
- struct strbuf *temp_tab,
- struct reftable_log_expiry_config *config)
+ struct reftable_log_expiry_config *config,
+ struct tempfile **temp_table_out)
{
struct strbuf next_name = STRBUF_INIT;
- int tab_fd = -1;
+ struct strbuf table_path = STRBUF_INIT;
struct reftable_writer *wr = NULL;
+ struct tempfile *temp_table;
+ int temp_table_fd;
int err = 0;
format_name(&next_name,
reftable_reader_min_update_index(st->readers[first]),
reftable_reader_max_update_index(st->readers[last]));
+ stack_filename(&table_path, st, next_name.buf);
+ strbuf_addstr(&table_path, ".temp.XXXXXX");
- stack_filename(temp_tab, st, next_name.buf);
- strbuf_addstr(temp_tab, ".temp.XXXXXX");
+ temp_table = mks_tempfile(table_path.buf);
+ if (!temp_table) {
+ err = REFTABLE_IO_ERROR;
+ goto done;
+ }
+ temp_table_fd = get_tempfile_fd(temp_table);
- tab_fd = mkstemp(temp_tab->buf);
if (st->config.default_permissions &&
- chmod(temp_tab->buf, st->config.default_permissions) < 0) {
+ chmod(get_tempfile_path(temp_table), st->config.default_permissions) < 0) {
err = REFTABLE_IO_ERROR;
goto done;
}
- wr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd, &st->config);
-
+ wr = reftable_new_writer(reftable_fd_write, reftable_fd_flush,
+ &temp_table_fd, &st->config);
err = stack_write_compact(st, wr, first, last, config);
if (err < 0)
goto done;
+
err = reftable_writer_close(wr);
if (err < 0)
goto done;
- err = close(tab_fd);
- tab_fd = 0;
+ err = close_tempfile_gently(temp_table);
+ if (err < 0)
+ goto done;
+
+ *temp_table_out = temp_table;
+ temp_table = NULL;
done:
+ delete_tempfile(&temp_table);
reftable_writer_free(wr);
- if (tab_fd > 0) {
- close(tab_fd);
- tab_fd = 0;
- }
- if (err != 0 && temp_tab->len > 0) {
- unlink(temp_tab->buf);
- strbuf_release(temp_tab);
- }
strbuf_release(&next_name);
+ strbuf_release(&table_path);
return err;
}
@@ -979,12 +985,12 @@ static int stack_compact_range(struct reftable_stack *st,
struct reftable_log_expiry_config *expiry)
{
struct strbuf tables_list_buf = STRBUF_INIT;
- struct strbuf new_table_temp_path = STRBUF_INIT;
struct strbuf new_table_name = STRBUF_INIT;
struct strbuf new_table_path = STRBUF_INIT;
struct strbuf table_name = STRBUF_INIT;
struct lock_file tables_list_lock = LOCK_INIT;
struct lock_file *table_locks = NULL;
+ struct tempfile *new_table = NULL;
int is_empty_table = 0, err = 0;
size_t i;
@@ -1059,7 +1065,7 @@ static int stack_compact_range(struct reftable_stack *st,
* these tables may end up with an empty new table in case tombstones
* end up cancelling out all refs in that range.
*/
- err = stack_compact_locked(st, first, last, &new_table_temp_path, expiry);
+ err = stack_compact_locked(st, first, last, expiry, &new_table);
if (err < 0) {
if (err != REFTABLE_EMPTY_TABLE_ERROR)
goto done;
@@ -1099,7 +1105,7 @@ static int stack_compact_range(struct reftable_stack *st,
strbuf_addstr(&new_table_name, ".ref");
stack_filename(&new_table_path, st, new_table_name.buf);
- err = rename(new_table_temp_path.buf, new_table_path.buf);
+ err = rename_tempfile(&new_table, new_table_path.buf);
if (err < 0) {
err = REFTABLE_IO_ERROR;
goto done;
@@ -1166,9 +1172,10 @@ static int stack_compact_range(struct reftable_stack *st,
rollback_lock_file(&table_locks[i - first]);
reftable_free(table_locks);
+ delete_tempfile(&new_table);
strbuf_release(&new_table_name);
strbuf_release(&new_table_path);
- strbuf_release(&new_table_temp_path);
+
strbuf_release(&tables_list_buf);
strbuf_release(&table_name);
return err;
--
2.44.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related
* Re: [PATCH v4 1/1] [PATCH] t9117: prefer test_path_* helper functions
From: shejialuo @ 2024-03-04 11:45 UTC (permalink / raw)
To: Patrick Steinhardt; +Cc: Eric Sunshine, Junio C Hamano, git
In-Reply-To: <ZeWbdvFmhUYN9ekE@tanuki>
Thanks for your advice. I will remember this suggestion.
^ permalink raw reply
* Re: [PATCH] SoC 2024: clarify `test_path_is_*` conversion microproject
From: Christian Couder @ 2024-03-04 13:42 UTC (permalink / raw)
To: Patrick Steinhardt; +Cc: Junio C Hamano, git, Kaartic Sivaraam
In-Reply-To: <84995a068640c72c8f17406ffa0441c7fdba4bdc.1709543804.git.ps@pks.im>
On Mon, Mar 4, 2024 at 10:17 AM Patrick Steinhardt <ps@pks.im> wrote:
>
> One of our proposed microprojects is to convert instances of `test -e`
> and related functions to instead use `test_path_exists` or similar. This
> conversion is only feasible when `test -e` is not used as part of a
> control statement, as the replacement is used to _assert_ a condition
> instead of merely testing for it.
>
> Clarify the microproject's description accordingly.
Applied and pushed, thanks!
I wonder if it would be better to create a PR in
https://github.com/git/git.github.io/ and perhaps just send a link to
it, rather than sending patches to the mailing list, as patches on the
mailing list could be mistaken by tools and perhaps people as applying
to the Git code base.
^ permalink raw reply
* [PATCH v2] setup: remove unnecessary variable
From: Ghanshyam Thakkar @ 2024-03-04 15:18 UTC (permalink / raw)
To: shyamthakkar001
Cc: christian.couder, git, gitster, johannes.schindelin, newren
In-Reply-To: <20240229134114.285393-2-shyamthakkar001@gmail.com>
The TODO comment suggested to heed core.bare from template config file
if no command line override given. And the prev_bare_repository
variable seems to have been placed for this sole purpose as it is not
used anywhere else.
However, it was clarified by Junio [1] that such values (including
core.bare) are ignored intentionally and does not make sense to
propagate them from template config to repository config. Also, the
directories for the worktree and repository are already created, and
therefore the bare/non-bare decision has already been made, by the
point we reach the codepath where the TODO comment is placed.
Therefore, prev_bare_repository does not have a usecase with/without
supporting core.bare from template. And the removal of
prev_bare_repository is safe as proved by the later part of the
comment:
"Unfortunately, the line above is equivalent to
is_bare_repository_cfg = !work_tree;
which ignores the config entirely even if no `--[no-]bare`
command line option was present.
To see why, note that before this function, there was this call:
prev_bare_repository = is_bare_repository()
expanding the right hand side:
= is_bare_repository_cfg && !get_git_work_tree()
= is_bare_repository_cfg && !work_tree
note that the last simplification above is valid because nothing
calls repo_init() or set_git_work_tree() between any of the
relevant calls in the code, and thus the !get_git_work_tree()
calls will return the same result each time. So, what we are
interested in computing is the right hand side of the line of
code just above this comment:
prev_bare_repository || !work_tree
= is_bare_repository_cfg && !work_tree || !work_tree
= !work_tree
because "A && !B || !B == !B" for all boolean values of A & B."
Therefore, remove the TODO comment and remove prev_bare_repository
variable. Also, update relevant testcases and remove one redundant
testcase.
[1]: https://lore.kernel.org/git/xmqqjzonpy9l.fsf@gitster.g/
Helped-by: Elijah Newren <newren@gmail.com>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Ghanshyam Thakkar <shyamthakkar001@gmail.com>
---
setup.c | 36 +++---------------------------------
t/t1301-shared-repo.sh | 15 ++-------------
t/t5606-clone-options.sh | 6 +++---
3 files changed, 8 insertions(+), 49 deletions(-)
diff --git a/setup.c b/setup.c
index b69b1cbc2a..81accd1213 100644
--- a/setup.c
+++ b/setup.c
@@ -1961,7 +1961,6 @@ void create_reference_database(unsigned int ref_storage_format,
static int create_default_files(const char *template_path,
const char *original_git_dir,
const struct repository_format *fmt,
- int prev_bare_repository,
int init_shared_repository)
{
struct stat st1;
@@ -1996,34 +1995,8 @@ static int create_default_files(const char *template_path,
*/
if (init_shared_repository != -1)
set_shared_repository(init_shared_repository);
- /*
- * TODO: heed core.bare from config file in templates if no
- * command-line override given
- */
- is_bare_repository_cfg = prev_bare_repository || !work_tree;
- /* TODO (continued):
- *
- * Unfortunately, the line above is equivalent to
- * is_bare_repository_cfg = !work_tree;
- * which ignores the config entirely even if no `--[no-]bare`
- * command line option was present.
- *
- * To see why, note that before this function, there was this call:
- * prev_bare_repository = is_bare_repository()
- * expanding the right hand side:
- * = is_bare_repository_cfg && !get_git_work_tree()
- * = is_bare_repository_cfg && !work_tree
- * note that the last simplification above is valid because nothing
- * calls repo_init() or set_git_work_tree() between any of the
- * relevant calls in the code, and thus the !get_git_work_tree()
- * calls will return the same result each time. So, what we are
- * interested in computing is the right hand side of the line of
- * code just above this comment:
- * prev_bare_repository || !work_tree
- * = is_bare_repository_cfg && !work_tree || !work_tree
- * = !work_tree
- * because "A && !B || !B == !B" for all boolean values of A & B.
- */
+
+ is_bare_repository_cfg = !work_tree;
/*
* We would have created the above under user's umask -- under
@@ -2175,7 +2148,6 @@ int init_db(const char *git_dir, const char *real_git_dir,
int exist_ok = flags & INIT_DB_EXIST_OK;
char *original_git_dir = real_pathdup(git_dir, 1);
struct repository_format repo_fmt = REPOSITORY_FORMAT_INIT;
- int prev_bare_repository;
if (real_git_dir) {
struct stat st;
@@ -2201,7 +2173,6 @@ int init_db(const char *git_dir, const char *real_git_dir,
safe_create_dir(git_dir, 0);
- prev_bare_repository = is_bare_repository();
/* Check to see if the repository version is right.
* Note that a newly created repository does not have
@@ -2214,8 +2185,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
validate_ref_storage_format(&repo_fmt, ref_storage_format);
reinit = create_default_files(template_dir, original_git_dir,
- &repo_fmt, prev_bare_repository,
- init_shared_repository);
+ &repo_fmt, init_shared_repository);
/*
* Now that we have set up both the hash algorithm and the ref storage
diff --git a/t/t1301-shared-repo.sh b/t/t1301-shared-repo.sh
index b1eb5c01b8..29cf8a9661 100755
--- a/t/t1301-shared-repo.sh
+++ b/t/t1301-shared-repo.sh
@@ -52,7 +52,7 @@ test_expect_success 'shared=all' '
test 2 = $(git config core.sharedrepository)
'
-test_expect_failure 'template can set core.bare' '
+test_expect_success 'template cannot set core.bare' '
test_when_finished "rm -rf subdir" &&
test_when_finished "rm -rf templates" &&
test_config core.bare true &&
@@ -60,18 +60,7 @@ test_expect_failure 'template can set core.bare' '
mkdir -p templates/ &&
cp .git/config templates/config &&
git init --template=templates subdir &&
- test_path_exists subdir/HEAD
-'
-
-test_expect_success 'template can set core.bare but overridden by command line' '
- test_when_finished "rm -rf subdir" &&
- test_when_finished "rm -rf templates" &&
- test_config core.bare true &&
- umask 0022 &&
- mkdir -p templates/ &&
- cp .git/config templates/config &&
- git init --no-bare --template=templates subdir &&
- test_path_exists subdir/.git/HEAD
+ test_path_is_missing subdir/HEAD
'
test_expect_success POSIXPERM 'update-server-info honors core.sharedRepository' '
diff --git a/t/t5606-clone-options.sh b/t/t5606-clone-options.sh
index a400bcca62..e93e0d0cc3 100755
--- a/t/t5606-clone-options.sh
+++ b/t/t5606-clone-options.sh
@@ -120,14 +120,14 @@ test_expect_success 'prefers -c config over --template config' '
'
-test_expect_failure 'prefers --template config even for core.bare' '
+test_expect_success 'ignore --template config for core.bare' '
template="$TRASH_DIRECTORY/template-with-bare-config" &&
mkdir "$template" &&
git config --file "$template/config" core.bare true &&
git clone "--template=$template" parent clone-bare-config &&
- test "$(git -C clone-bare-config config --local core.bare)" = "true" &&
- test_path_is_file clone-bare-config/HEAD
+ test "$(git -C clone-bare-config config --local core.bare)" = "false" &&
+ test_path_is_missing clone-bare-config/HEAD
'
test_expect_success 'prefers config "clone.defaultRemoteName" over default' '
--
2.44.0
^ permalink raw reply related
* [PATCH 0/4] trace2: move 'def_param' events into 'cmd_name' and 'cmd_alias'
From: Jeff Hostetler via GitGitGadget @ 2024-03-04 15:40 UTC (permalink / raw)
To: git; +Cc: Jeff Hostetler
Some Git commands do not emit def_param events for interesting config and
environment variable settings. Let's fix that.
Builtin commands compiled into git.c have the normal control flow and emit a
cmd_name event and then def_param events for each interesting config and
environment variable. However, some special "query" commands, like
--exec-path, or some forms of alias expansion, emitted a cmd_name but did
not emit def_param events.
Also, special commands git-remote-https is built from remote-curl.c and
git-http-fetch is built from http-fetch.c and do not use the normal set up
in git.c. These emitted a cmd_name but not def_param events.
To minimize the footprint of this commit, move the calls to
trace2_cmd_list_config() and trace2_cmd_list_env_vars() into
trace2_cmd_name() and trace2_cmd_alias() so that we always get a set
def_param events when a cmd_name or cmd_alias event is generated.
Users can define local config settings on a repo to classify/name a repo
(e.g. "project-foo" vs "personal") and use the def_param feature to label
Trace2 data so that (a third-party) telemetry service does not collect data
on personal repos or so that telemetry from one work repo is distinguishable
from another work repo.
Jeff Hostetler (4):
t0211: demonstrate missing 'def_param' events for certain commands
trace2: avoid emitting 'def_param' set more than once
trace2: emit 'def_param' set with 'cmd_name' event
trace2: remove unneeded calls to generate 'def_param' set
git.c | 6 --
t/t0211-trace2-perf.sh | 231 +++++++++++++++++++++++++++++++++++++++++
trace2.c | 15 +++
3 files changed, 246 insertions(+), 6 deletions(-)
base-commit: 0f9d4d28b7e6021b7e6db192b7bf47bd3a0d0d1d
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1679%2Fjeffhostetler%2Falways-emit-def-param-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1679/jeffhostetler/always-emit-def-param-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1679
--
gitgitgadget
^ permalink raw reply
* [PATCH 1/4] t0211: demonstrate missing 'def_param' events for certain commands
From: Jeff Hostetler via GitGitGadget @ 2024-03-04 15:40 UTC (permalink / raw)
To: git; +Cc: Jeff Hostetler, Jeff Hostetler
In-Reply-To: <pull.1679.git.1709566808.gitgitgadget@gmail.com>
From: Jeff Hostetler <jeffhostetler@github.com>
Some Git commands fail to emit 'def_param' events for interesting
config and environment variable settings.
Add unit tests to demonstrate this.
Most commands are considered "builtin" and are based upon git.c.
These typically do emit 'def_param' events. Exceptions are some of
the "query" commands, the "run-dashed" mechanism, and alias handling.
Commands built from remote-curl.c (instead of git.c), such as
"git-remote-https", do not emit 'def_param' events.
Likewise, "git-http-fetch" is built http-fetch.c and does not emit
them.
Signed-off-by: Jeff Hostetler <jeffhostetler@github.com>
---
t/t0211-trace2-perf.sh | 231 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 231 insertions(+)
diff --git a/t/t0211-trace2-perf.sh b/t/t0211-trace2-perf.sh
index 290b6eaaab1..588c5bad033 100755
--- a/t/t0211-trace2-perf.sh
+++ b/t/t0211-trace2-perf.sh
@@ -287,4 +287,235 @@ test_expect_success 'unsafe URLs are redacted by default' '
grep "d0|main|def_param|.*|remote.origin.url:https://user:pwd@example.com" actual
'
+# Confirm that the requested command produces a "cmd_name" and a
+# set of "def_param" events.
+#
+try_simple () {
+ test_when_finished "rm prop.perf actual" &&
+
+ cmd=$1 &&
+ cmd_name=$2 &&
+
+ test_config_global "trace2.configParams" "cfg.prop.*" &&
+ test_config_global "trace2.envvars" "ENV_PROP_FOO,ENV_PROP_BAR" &&
+
+ test_config_global "cfg.prop.foo" "red" &&
+
+ ENV_PROP_FOO=blue \
+ GIT_TRACE2_PERF="$(pwd)/prop.perf" \
+ $cmd &&
+ perl "$TEST_DIRECTORY/t0211/scrub_perf.perl" <prop.perf >actual &&
+ grep "d0|main|cmd_name|.*|$cmd_name" actual &&
+ grep "d0|main|def_param|.*|cfg.prop.foo:red" actual &&
+ grep "d0|main|def_param|.*|ENV_PROP_FOO:blue" actual
+}
+
+# Representative mainstream builtin Git command dispatched
+# in run_builtin() in git.c
+#
+test_expect_success 'expect def_params for normal builtin command' '
+ try_simple "git version" "version"
+'
+
+# Representative query command dispatched in handle_options()
+# in git.c
+#
+test_expect_failure 'expect def_params for query command' '
+ try_simple "git --man-path" "_query_"
+'
+
+# remote-curl.c does not use the builtin setup in git.c, so confirm
+# that executables built from remote-curl.c emit def_params.
+#
+# Also tests the dashed-command handling where "git foo" silently
+# spawns "git-foo". Make sure that both commands should emit
+# def_params.
+#
+# Pass bogus arguments to remote-https and allow the command to fail
+# because we don't actually have a remote to fetch from. We just want
+# to see the run-dashed code run an executable built from
+# remote-curl.c rather than git.c. Confirm that we get def_param
+# events from both layers.
+#
+test_expect_failure 'expect def_params for remote-curl and _run_dashed_' '
+ test_when_finished "rm prop.perf actual" &&
+
+ test_config_global "trace2.configParams" "cfg.prop.*" &&
+ test_config_global "trace2.envvars" "ENV_PROP_FOO,ENV_PROP_BAR" &&
+
+ test_config_global "cfg.prop.foo" "red" &&
+
+ test_might_fail env \
+ ENV_PROP_FOO=blue \
+ GIT_TRACE2_PERF="$(pwd)/prop.perf" \
+ git remote-http x y &&
+
+ perl "$TEST_DIRECTORY/t0211/scrub_perf.perl" <prop.perf >actual &&
+
+ grep "d0|main|cmd_name|.*|_run_dashed_" actual &&
+ grep "d0|main|def_param|.*|cfg.prop.foo:red" actual &&
+ grep "d0|main|def_param|.*|ENV_PROP_FOO:blue" actual &&
+
+ grep "d1|main|cmd_name|.*|remote-curl" actual &&
+ grep "d1|main|def_param|.*|cfg.prop.foo:red" actual &&
+ grep "d1|main|def_param|.*|ENV_PROP_FOO:blue" actual
+'
+
+# Similarly, `git-http-fetch` is not built from git.c so do a
+# trivial fetch so that the main git.c run-dashed code spawns
+# an executable built from http-fetch.c. Confirm that we get
+# def_param events from both layers.
+#
+test_expect_failure 'expect def_params for http-fetch and _run_dashed_' '
+ test_when_finished "rm prop.perf actual" &&
+
+ test_config_global "trace2.configParams" "cfg.prop.*" &&
+ test_config_global "trace2.envvars" "ENV_PROP_FOO,ENV_PROP_BAR" &&
+
+ test_config_global "cfg.prop.foo" "red" &&
+
+ test_might_fail env \
+ ENV_PROP_FOO=blue \
+ GIT_TRACE2_PERF="$(pwd)/prop.perf" \
+ git http-fetch --stdin file:/// <<-EOF &&
+ EOF
+
+ perl "$TEST_DIRECTORY/t0211/scrub_perf.perl" <prop.perf >actual &&
+
+ grep "d0|main|cmd_name|.*|_run_dashed_" actual &&
+ grep "d0|main|def_param|.*|cfg.prop.foo:red" actual &&
+ grep "d0|main|def_param|.*|ENV_PROP_FOO:blue" actual &&
+
+ grep "d1|main|cmd_name|.*|http-fetch" actual &&
+ grep "d1|main|def_param|.*|cfg.prop.foo:red" actual &&
+ grep "d1|main|def_param|.*|ENV_PROP_FOO:blue" actual
+'
+
+# Historically, alias expansion explicitly emitted the def_param
+# events (independent of whether the command was a builtin, a Git
+# command or arbitrary shell command) so that it wasn't dependent
+# upon the unpeeling of the alias. Let's make sure that we preserve
+# the net effect.
+#
+test_expect_success 'expect def_params during git alias expansion' '
+ test_when_finished "rm prop.perf actual" &&
+
+ test_config_global "trace2.configParams" "cfg.prop.*" &&
+ test_config_global "trace2.envvars" "ENV_PROP_FOO,ENV_PROP_BAR" &&
+
+ test_config_global "cfg.prop.foo" "red" &&
+
+ test_config_global "alias.xxx" "version" &&
+
+ ENV_PROP_FOO=blue \
+ GIT_TRACE2_PERF="$(pwd)/prop.perf" \
+ git xxx &&
+
+ perl "$TEST_DIRECTORY/t0211/scrub_perf.perl" <prop.perf >actual &&
+
+ # "git xxx" is first mapped to "git-xxx" and the child will fail.
+ grep "d0|main|cmd_name|.*|_run_dashed_ (_run_dashed_)" actual &&
+
+ # We unpeel that and substitute "version" into "xxx" (giving
+ # "git version") and update the cmd_name event.
+ grep "d0|main|cmd_name|.*|_run_git_alias_ (_run_dashed_/_run_git_alias_)" actual &&
+
+ # These def_param events could be associated with either of the
+ # above cmd_name events. It does not matter.
+ grep "d0|main|def_param|.*|cfg.prop.foo:red" actual &&
+ grep "d0|main|def_param|.*|ENV_PROP_FOO:blue" actual &&
+
+ # The "git version" child sees a different cmd_name hierarchy.
+ # Also test the def_param (only for completeness).
+ grep "d1|main|cmd_name|.*|version (_run_dashed_/_run_git_alias_/version)" actual &&
+ grep "d1|main|def_param|.*|cfg.prop.foo:red" actual &&
+ grep "d1|main|def_param|.*|ENV_PROP_FOO:blue" actual
+'
+
+test_expect_success 'expect def_params during shell alias expansion' '
+ test_when_finished "rm prop.perf actual" &&
+
+ test_config_global "trace2.configParams" "cfg.prop.*" &&
+ test_config_global "trace2.envvars" "ENV_PROP_FOO,ENV_PROP_BAR" &&
+
+ test_config_global "cfg.prop.foo" "red" &&
+
+ test_config_global "alias.xxx" "!git version" &&
+
+ ENV_PROP_FOO=blue \
+ GIT_TRACE2_PERF="$(pwd)/prop.perf" \
+ git xxx &&
+
+ perl "$TEST_DIRECTORY/t0211/scrub_perf.perl" <prop.perf >actual &&
+
+ # "git xxx" is first mapped to "git-xxx" and the child will fail.
+ grep "d0|main|cmd_name|.*|_run_dashed_ (_run_dashed_)" actual &&
+
+ # We unpeel that and substitute "git version" for "git xxx" (as a
+ # shell command. Another cmd_name event is emitted as we unpeel.
+ grep "d0|main|cmd_name|.*|_run_shell_alias_ (_run_dashed_/_run_shell_alias_)" actual &&
+
+ # These def_param events could be associated with either of the
+ # above cmd_name events. It does not matter.
+ grep "d0|main|def_param|.*|cfg.prop.foo:red" actual &&
+ grep "d0|main|def_param|.*|ENV_PROP_FOO:blue" actual &&
+
+ # We get the following only because we used a git command for the
+ # shell command. In general, it could have been a shell script and
+ # we would see nothing.
+ #
+ # The child knows the cmd_name hierarchy so it includes it.
+ grep "d1|main|cmd_name|.*|version (_run_dashed_/_run_shell_alias_/version)" actual &&
+ grep "d1|main|def_param|.*|cfg.prop.foo:red" actual &&
+ grep "d1|main|def_param|.*|ENV_PROP_FOO:blue" actual
+'
+
+test_expect_failure 'expect def_params during nested git alias expansion' '
+ test_when_finished "rm prop.perf actual" &&
+
+ test_config_global "trace2.configParams" "cfg.prop.*" &&
+ test_config_global "trace2.envvars" "ENV_PROP_FOO,ENV_PROP_BAR" &&
+
+ test_config_global "cfg.prop.foo" "red" &&
+
+ test_config_global "alias.xxx" "yyy" &&
+ test_config_global "alias.yyy" "version" &&
+
+ ENV_PROP_FOO=blue \
+ GIT_TRACE2_PERF="$(pwd)/prop.perf" \
+ git xxx &&
+
+ perl "$TEST_DIRECTORY/t0211/scrub_perf.perl" <prop.perf >actual &&
+
+ # "git xxx" is first mapped to "git-xxx" and try to spawn "git-xxx"
+ # and the child will fail.
+ grep "d0|main|cmd_name|.*|_run_dashed_ (_run_dashed_)" actual &&
+ grep "d0|main|child_start|.*|.* class:dashed argv:\[git-xxx\]" actual &&
+
+ # We unpeel that and substitute "yyy" into "xxx" (giving "git yyy")
+ # and spawn "git-yyy" and the child will fail.
+ grep "d0|main|alias|.*|alias:xxx argv:\[yyy\]" actual &&
+ grep "d0|main|cmd_name|.*|_run_dashed_ (_run_dashed_/_run_dashed_)" actual &&
+ grep "d0|main|child_start|.*|.* class:dashed argv:\[git-yyy\]" actual &&
+
+ # We unpeel that and substitute "version" into "xxx" (giving
+ # "git version") and update the cmd_name event.
+ grep "d0|main|alias|.*|alias:yyy argv:\[version\]" actual &&
+ grep "d0|main|cmd_name|.*|_run_git_alias_ (_run_dashed_/_run_dashed_/_run_git_alias_)" actual &&
+
+ # These def_param events could be associated with any of the
+ # above cmd_name events. It does not matter.
+ grep "d0|main|def_param|.*|cfg.prop.foo:red" actual >actual.matches &&
+ grep "d0|main|def_param|.*|ENV_PROP_FOO:blue" actual &&
+
+ # However, we do not want them repeated each time we unpeel.
+ test_line_count = 1 actual.matches &&
+
+ # The "git version" child sees a different cmd_name hierarchy.
+ # Also test the def_param (only for completeness).
+ grep "d1|main|cmd_name|.*|version (_run_dashed_/_run_dashed_/_run_git_alias_/version)" actual &&
+ grep "d1|main|def_param|.*|cfg.prop.foo:red" actual &&
+ grep "d1|main|def_param|.*|ENV_PROP_FOO:blue" actual
+'
+
test_done
--
gitgitgadget
^ permalink raw reply related
* [PATCH 2/4] trace2: avoid emitting 'def_param' set more than once
From: Jeff Hostetler via GitGitGadget @ 2024-03-04 15:40 UTC (permalink / raw)
To: git; +Cc: Jeff Hostetler, Jeff Hostetler
In-Reply-To: <pull.1679.git.1709566808.gitgitgadget@gmail.com>
From: Jeff Hostetler <jeffhostetler@github.com>
During nested alias expansion it is possible for
"trace2_cmd_list_config()" and "trace2_cmd_list_env_vars()"
to be called more than once. This causes a full set of
'def_param' events to be emitted each time. Let's avoid
that.
Add code to those two functions to only emit them once.
Signed-off-by: Jeff Hostetler <jeffhostetler@github.com>
---
t/t0211-trace2-perf.sh | 2 +-
trace2.c | 12 ++++++++++++
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/t/t0211-trace2-perf.sh b/t/t0211-trace2-perf.sh
index 588c5bad033..7b353195396 100755
--- a/t/t0211-trace2-perf.sh
+++ b/t/t0211-trace2-perf.sh
@@ -470,7 +470,7 @@ test_expect_success 'expect def_params during shell alias expansion' '
grep "d1|main|def_param|.*|ENV_PROP_FOO:blue" actual
'
-test_expect_failure 'expect def_params during nested git alias expansion' '
+test_expect_success 'expect def_params during nested git alias expansion' '
test_when_finished "rm prop.perf actual" &&
test_config_global "trace2.configParams" "cfg.prop.*" &&
diff --git a/trace2.c b/trace2.c
index f1e268bd159..facce641ef3 100644
--- a/trace2.c
+++ b/trace2.c
@@ -464,17 +464,29 @@ void trace2_cmd_alias_fl(const char *file, int line, const char *alias,
void trace2_cmd_list_config_fl(const char *file, int line)
{
+ static int emitted = 0;
+
if (!trace2_enabled)
return;
+ if (emitted)
+ return;
+ emitted = 1;
+
tr2_cfg_list_config_fl(file, line);
}
void trace2_cmd_list_env_vars_fl(const char *file, int line)
{
+ static int emitted = 0;
+
if (!trace2_enabled)
return;
+ if (emitted)
+ return;
+ emitted = 1;
+
tr2_list_env_vars_fl(file, line);
}
--
gitgitgadget
^ permalink raw reply related
* [PATCH 3/4] trace2: emit 'def_param' set with 'cmd_name' event
From: Jeff Hostetler via GitGitGadget @ 2024-03-04 15:40 UTC (permalink / raw)
To: git; +Cc: Jeff Hostetler, Jeff Hostetler
In-Reply-To: <pull.1679.git.1709566808.gitgitgadget@gmail.com>
From: Jeff Hostetler <jeffhostetler@github.com>
Some commands do not cause a set of 'def_param' events to be emitted.
This includes "git-remote-https", "git-http-fetch", and various
"query" commands, like "git --man-path".
Since all of these commands do emit a 'cmd_name' event, add code to
the "trace2_cmd_name()" function to generate the set of 'def_param'
events.
We can later remove explicit calls to "trace2_cmd_list_config()" and
"trace2_cmd_list_env_vars()" in git.c.
Signed-off-by: Jeff Hostetler <jeffhostetler@github.com>
---
t/t0211-trace2-perf.sh | 6 +++---
trace2.c | 3 +++
2 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/t/t0211-trace2-perf.sh b/t/t0211-trace2-perf.sh
index 7b353195396..13ef69b92f8 100755
--- a/t/t0211-trace2-perf.sh
+++ b/t/t0211-trace2-perf.sh
@@ -320,7 +320,7 @@ test_expect_success 'expect def_params for normal builtin command' '
# Representative query command dispatched in handle_options()
# in git.c
#
-test_expect_failure 'expect def_params for query command' '
+test_expect_success 'expect def_params for query command' '
try_simple "git --man-path" "_query_"
'
@@ -337,7 +337,7 @@ test_expect_failure 'expect def_params for query command' '
# remote-curl.c rather than git.c. Confirm that we get def_param
# events from both layers.
#
-test_expect_failure 'expect def_params for remote-curl and _run_dashed_' '
+test_expect_success 'expect def_params for remote-curl and _run_dashed_' '
test_when_finished "rm prop.perf actual" &&
test_config_global "trace2.configParams" "cfg.prop.*" &&
@@ -366,7 +366,7 @@ test_expect_failure 'expect def_params for remote-curl and _run_dashed_' '
# an executable built from http-fetch.c. Confirm that we get
# def_param events from both layers.
#
-test_expect_failure 'expect def_params for http-fetch and _run_dashed_' '
+test_expect_success 'expect def_params for http-fetch and _run_dashed_' '
test_when_finished "rm prop.perf actual" &&
test_config_global "trace2.configParams" "cfg.prop.*" &&
diff --git a/trace2.c b/trace2.c
index facce641ef3..f894532d053 100644
--- a/trace2.c
+++ b/trace2.c
@@ -433,6 +433,9 @@ void trace2_cmd_name_fl(const char *file, int line, const char *name)
for_each_wanted_builtin (j, tgt_j)
if (tgt_j->pfn_command_name_fl)
tgt_j->pfn_command_name_fl(file, line, name, hierarchy);
+
+ trace2_cmd_list_config();
+ trace2_cmd_list_env_vars();
}
void trace2_cmd_mode_fl(const char *file, int line, const char *mode)
--
gitgitgadget
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox