[PATCH] odb/files: be less aggressive with geometric repacking

[PATCH] odb/files: be less aggressive with geometric repacking

[Patrick Steinhardt]

When performing auto-maintenance with geometric repacking we have two
conditions that may trigger a repack:

  - Either the geometric sequence of packfiles is invalidated.

  - Or we have too many loose objects.

The first condition shouldn't trigger all that often: it may be hit when
we fetch a new packfile, but users tend to not do that all the time. The
second condition is what typically triggers more regularly though, as
every command that ends up writing new objects may cause us to cross the
threshold of loose objects. It is thus preferable to not be too
aggressive here, as otherwise we may end up repacking objects quite
often.

For the geometric-repacking strategy though we have a default of 100
objects, only. As we're approximating the count of objects by only
reading the "objects/17/" shared, we'd only need 2 objects in there
before we perform a repack by default, which is quite aggressive.
git-gc(1) on the other hand has a default of 6700, so it is quite a bit
more conservative here.

Being this aggressive is also causing problems as reported by our users.
When running lots of concurrent writers, those writes will constantly
end up spawning maintenance jobs that end up repacking objects. As we
also prune objects, a concurrently running process that tries to write
an object may see that the sharding directories get removed under their
feet. While we try re-creating such leading directories, we only do so a
single time, and it may happen that the directory vanishes again before
we had the chance to create the loose object. This is not a new problem,
but it is exacerbated by us running maintenance this aggressively.

Improve the status quo by reducing the frequency at which we pack loose
objects to the same frequency that git-gc(1) uses.

Reported-by: Stefan Haller <lists@haller-berlin.de>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
Hi,

as reported by Stefan at [1]. Thanks!

Patrick

[1]: <4f6a96ac-d993-4872-b3c4-30d899f61ca9@haller-berlin.de>
---
 Documentation/config/maintenance.adoc | 2 +-
 odb/source-files.c                    | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/Documentation/config/maintenance.adoc b/Documentation/config/maintenance.adoc
index b578856dde..da8be9f812 100644
--- a/Documentation/config/maintenance.adoc
+++ b/Documentation/config/maintenance.adoc
@@ -101,7 +101,7 @@ maintenance.geometric-repack.auto::
 	there are packfiles that need to be merged together to retain the
 	geometric progression, or when there are at least this many loose
 	objects that would be written into a new packfile. The default value is
-	100.
+	6700.
 
 maintenance.geometric-repack.splitFactor::
 	This integer config option controls the factor used for the geometric
diff --git a/odb/source-files.c b/odb/source-files.c
index 5a68af7d84..555e466145 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -521,7 +521,7 @@ bool odb_source_files_optimize_required(struct odb_source *source,
 		};
 		struct existing_packs existing_packs = EXISTING_PACKS_INIT;
 		struct string_list kept_packs = STRING_LIST_INIT_DUP;
-		int auto_value = 100;
+		int auto_value = 6700;
 		bool ret;
 
 		repo_config_get_int(repo, "maintenance.geometric-repack.auto",

---
base-commit: 010afd3166ddc64c9863b1506f12cbcdda0d4ea1
change-id: 20260810-pks-geometric-maintenance-reduce-frequency-5c1c9423ceb3

Re: [PATCH] odb/files: be less aggressive with geometric repacking

[Justin Tobler]

On 26/08/11 11:04AM, Patrick Steinhardt wrote:
> When performing auto-maintenance with geometric repacking we have two
> conditions that may trigger a repack:
> 
>   - Either the geometric sequence of packfiles is invalidated.
> 
>   - Or we have too many loose objects.
> 
> The first condition shouldn't trigger all that often: it may be hit when
> we fetch a new packfile, but users tend to not do that all the time. The
> second condition is what typically triggers more regularly though, as
> every command that ends up writing new objects may cause us to cross the
> threshold of loose objects. It is thus preferable to not be too
> aggressive here, as otherwise we may end up repacking objects quite
> often.
> 
> For the geometric-repacking strategy though we have a default of 100
> objects, only. As we're approximating the count of objects by only
> reading the "objects/17/" shared, we'd only need 2 objects in there
> before we perform a repack by default, which is quite aggressive.
> git-gc(1) on the other hand has a default of 6700, so it is quite a bit
> more conservative here.

Ok IIUC, the reason two loose objects can potentially trigger repacking
is because the heuristic used to estimate the number of loose objects
only counts objects in "objects/17/" and multiples it by 256 (the
maximum number of directories that are fanned-out). That makes sense and
indeed seems like it could lead to repacking processes be spawned more
frequently than desired.

My first thought is whether the heuristic itself should be updated to
capture a more accurate estimate for the number of objects. That would
of course require looking up more objects and thus be more expensive. If
the goal here is just for a very rough estimate anyways, maybe it
wouldn't be worth it though.

Increasing the loose object threshold here to be more conservative seems
like a reasonable approach. I'm not sure exactly why 6700 was chosen
here. 6700 / 256 ~= 26.2 which means "objects/17/" would have to contain
at least 27 objects before repacking is triggered. That is certainly
much more conservative. I see that 6700 has also been chosen else where
in the codebase as the threshold too. It might be nice to explain the
reasoning a bit more in the commit message though.

> Being this aggressive is also causing problems as reported by our users.
> When running lots of concurrent writers, those writes will constantly
> end up spawning maintenance jobs that end up repacking objects. As we
> also prune objects, a concurrently running process that tries to write
> an object may see that the sharding directories get removed under their
> feet. While we try re-creating such leading directories, we only do so a
> single time, and it may happen that the directory vanishes again before
> we had the chance to create the loose object. This is not a new problem,
> but it is exacerbated by us running maintenance this aggressively.
> 
> Improve the status quo by reducing the frequency at which we pack loose
> objects to the same frequency that git-gc(1) uses.

Makes sense and the patch itself looks trivially correct.

-Justin

Re: [PATCH] odb/files: be less aggressive with geometric repacking

[Patrick Steinhardt]

On Tue, Aug 11, 2026 at 03:44:12PM -0500, Justin Tobler wrote:
> On 26/08/11 11:04AM, Patrick Steinhardt wrote:
> > When performing auto-maintenance with geometric repacking we have two
> > conditions that may trigger a repack:
> > 
> >   - Either the geometric sequence of packfiles is invalidated.
> > 
> >   - Or we have too many loose objects.
> > 
> > The first condition shouldn't trigger all that often: it may be hit when
> > we fetch a new packfile, but users tend to not do that all the time. The
> > second condition is what typically triggers more regularly though, as
> > every command that ends up writing new objects may cause us to cross the
> > threshold of loose objects. It is thus preferable to not be too
> > aggressive here, as otherwise we may end up repacking objects quite
> > often.
> > 
> > For the geometric-repacking strategy though we have a default of 100
> > objects, only. As we're approximating the count of objects by only
> > reading the "objects/17/" shared, we'd only need 2 objects in there
> > before we perform a repack by default, which is quite aggressive.
> > git-gc(1) on the other hand has a default of 6700, so it is quite a bit
> > more conservative here.
> 
> Ok IIUC, the reason two loose objects can potentially trigger repacking
> is because the heuristic used to estimate the number of loose objects
> only counts objects in "objects/17/" and multiples it by 256 (the
> maximum number of directories that are fanned-out). That makes sense and
> indeed seems like it could lead to repacking processes be spawned more
> frequently than desired.
> 
> My first thought is whether the heuristic itself should be updated to
> capture a more accurate estimate for the number of objects. That would
> of course require looking up more objects and thus be more expensive. If
> the goal here is just for a very rough estimate anyways, maybe it
> wouldn't be worth it though.

That wouldn't really solve the problem though. The problem is not really
that the estimation can be wrong, it's rather that even if it was always
correct we're still being too aggressive with packing the loose objects.
Because ultimately, a 100 objects is a comparatively small threshold,
and leads to 67 times more repacking compared to git-gc(1).

> Increasing the loose object threshold here to be more conservative seems
> like a reasonable approach. I'm not sure exactly why 6700 was chosen
> here. 6700 / 256 ~= 26.2 which means "objects/17/" would have to contain
> at least 27 objects before repacking is triggered. That is certainly
> much more conservative. I see that 6700 has also been chosen else where
> in the codebase as the threshold too. It might be nice to explain the
> reasoning a bit more in the commit message though.

Hmm, don't I already do that? In the paragraph you're responding to I'm
saying that git-gc(1) already had that default forever, so I'm adjusting
our heuristic to match that.

Thanks!

Patrick