Slow git pack-refs --all

Slow git pack-refs --all

[Martin Fick]

I was hoping to get some help debugging a busy large repository where git pack-refs --all tends to regularly take over 5mins to run in production. As you can imagine this is particularly problematic on a busy Gerrit server since it tends to hold the packed-refs.lock file for most of this duration. Any help is greatly appreciated, see the details below.

-Martin

What did you do before the bug happened? (Steps to reproduce your issue)

I have a large repository (~90M objects, ~50GB, ~3M refs) which is regularly (every ~2hours) repacked and maintained, but generally gets at least 300+ updates per maintenance cycle.

What did you expect to happen? (Expected behavior)

git pack-refs --all to complete in under 20s when there are only 200 loose refs

What happened instead? (Actual behavior)

git pack-refs --all takes more than 3 minutes

What's different between what you expected and what actually happened?

This is much slower than expected

Anything else you want to add:

Although the packed-refs file is large, copying it takes less than 1s, so there isn't a writing throughput issue with the filesystem. Additionally, jgit can pack-refs --all in under 20s on the same repo, so I don't believe there is an issue locking the 200 loose refs either. When observing the filesystem, I do see the packed-refs.new growing at a rate that seems slower than expected as if much more is happening while writing this file, than just writing the file.

An strace shows about 200+ open("./objects..") calls interspersed between around ~26K write() calls. I am surprised to see pack-refs reading objects at all.

Although the repository is not in terrible shape before packing refs (~1500 loose objects, 37pack files). Surprisingly, repacking the repo first does speed it up so that packing refs then takes under 20s.

This repository is on NFS.

[System Info]
git version:
git version 2.45.2
cpu: x86_64
no commit associated with this build
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh
uname: Linux 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64
compiler info: gnuc: 14.1
libc info: glibc: 2.17
$SHELL (typically, interactive shell): /bin/bash


[Enabled Hooks]
not run from a git repository - no hooks to show

Re: Slow git pack-refs --all

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1864 bytes --]

On 2025-12-25 at 22:13:54, Martin Fick wrote:
> Although the packed-refs file is large, copying it takes less than 1s,
> so there isn't a writing throughput issue with the filesystem.
> Additionally, jgit can pack-refs --all in under 20s on the same repo,
> so I don't believe there is an issue locking the 200 loose refs
> either. When observing the filesystem, I do see the packed-refs.new
> growing at a rate that seems slower than expected as if much more is
> happening while writing this file, than just writing the file.
> 
> An strace shows about 200+ open("./objects..") calls interspersed
> between around ~26K write() calls. I am surprised to see pack-refs
> reading objects at all.

I think this is from `should_pack_ref`:

    /* Do not pack broken refs: */
    if (!ref_resolves_to_object(ref->name, refs->base.repo, ref->oid, ref->flags))
    	return 0;

So Git is going to need to verify that the object at least exists.  I
don't know why we would need to _open_ them, however.  Perhaps someone
else has ideas.

> Although the repository is not in terrible shape before packing refs
> (~1500 loose objects, 37pack files). Surprisingly, repacking the repo
> first does speed it up so that packing refs then takes under 20s.
> 
> This repository is on NFS.

That's almost certainly part of your performance problem, too.  Loading
a single pack file and index is going to be way, way faster than making
lots of network calls to open 37 pack file and 37 index files, plus at
least stat some loose objects.

I will note that at least some forges always have Git write pack files
and try to avoid loose objects altogether since that almost always
improves performance.  You may want to set `receive.unpackLimit` to 1 to
see if that helps in the general case.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: Slow git pack-refs --all

[Jeff King]

On Thu, Dec 25, 2025 at 11:38:30PM +0000, brian m. carlson wrote:

> I think this is from `should_pack_ref`:
> 
>     /* Do not pack broken refs: */
>     if (!ref_resolves_to_object(ref->name, refs->base.repo, ref->oid, ref->flags))
>     	return 0;
> 
> So Git is going to need to verify that the object at least exists.  I
> don't know why we would need to _open_ them, however.  Perhaps someone
> else has ideas.

The packed-refs file stores tag-peeling information. So pack-refs opens
the object for any newly written ref via peel_object(), which has to at
least read the header to get the type. That call happens via
write_with_updates() in packed-backend.c.

  If we wanted to be really pedantic, anything in refs/heads/ should not
  point to a non-commit and thus should never need to be peeled. I'm not
  sure if we want to embed that assumption in this code path, though
  (nor would it necessarily help Martin's case if the refs are not in
  refs/heads anyway).

-Peff

Re: Slow git pack-refs --all

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1579 bytes --]

On 2025-12-26 at 04:45:07, Jeff King wrote:
> On Thu, Dec 25, 2025 at 11:38:30PM +0000, brian m. carlson wrote:
> 
> > I think this is from `should_pack_ref`:
> > 
> >     /* Do not pack broken refs: */
> >     if (!ref_resolves_to_object(ref->name, refs->base.repo, ref->oid, ref->flags))
> >     	return 0;
> > 
> > So Git is going to need to verify that the object at least exists.  I
> > don't know why we would need to _open_ them, however.  Perhaps someone
> > else has ideas.
> 
> The packed-refs file stores tag-peeling information. So pack-refs opens
> the object for any newly written ref via peel_object(), which has to at
> least read the header to get the type. That call happens via
> write_with_updates() in packed-backend.c.
> 
>   If we wanted to be really pedantic, anything in refs/heads/ should not
>   point to a non-commit and thus should never need to be peeled. I'm not
>   sure if we want to embed that assumption in this code path, though
>   (nor would it necessarily help Martin's case if the refs are not in
>   refs/heads anyway).

I don't think that would be a good idea.  I know that people definitely
do updates of the loose refs by hand (although they should not) and so
it's entirely possible for them to contain invalid values, such as
having branches contain non-commit objects.

I wonder if reftable would avoid the need for this kind of expensive
check since it would already have the data peeled if need be and
wouldn't need to recompute the values.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: Slow git pack-refs --all

[Jeff King]

On Fri, Dec 26, 2025 at 05:15:31PM +0000, brian m. carlson wrote:

> >   If we wanted to be really pedantic, anything in refs/heads/ should not
> >   point to a non-commit and thus should never need to be peeled. I'm not
> >   sure if we want to embed that assumption in this code path, though
> >   (nor would it necessarily help Martin's case if the refs are not in
> >   refs/heads anyway).
> 
> I don't think that would be a good idea.  I know that people definitely
> do updates of the loose refs by hand (although they should not) and so
> it's entirely possible for them to contain invalid values, such as
> having branches contain non-commit objects.

Yeah, that matches my inclination.

> I wonder if reftable would avoid the need for this kind of expensive
> check since it would already have the data peeled if need be and
> wouldn't need to recompute the values.

It does the same amount of peeling, but it's amortized across more
operations (i.e., whatever did those ref updates in the first place)
rather than during the pack operation. And of course there really is no
pack operation per se with reftables, but I believe it avoids re-peeling
when rewriting entries during compaction.

It might actually do fewer object accesses overall if the ref-writing
operations have already loaded the objects in question (and thus it
knows whether they're tags or not, and may even have parsed tags in
memory). It can also do more in some cases (e.g., two loose writes will
peel for each write, whereas the files backend only bothers to peel
during packing).

-Peff

Re: Slow git pack-refs --all

[Martin Fick]

> From: Jeff King <peff@peff.net>
> Sent: Thursday, December 25, 2025 9:45 PM
>
> On Thu, Dec 25, 2025 at 11:38:30PM +0000, brian m. carlson wrote:
>> I think this is from `should_pack_ref`:
>>
>>     /* Do not pack broken refs: */
>>     if (!ref_resolves_to_object(ref->name, refs->base.repo, ref->oid, ref->flags))
>>       return 0;
>>
>> So Git is going to need to verify that the object at least exists.  I
>> don't know why we would need to _open_ them, however.  Perhaps someone
>> else has ideas.
>
>The packed-refs file stores tag-peeling information. So pack-refs opens
>the object for any newly written ref via peel_object(), which has to at
>least read the header to get the type. That call happens via
>write_with_updates() in packed-backend.c.

Thanks, this makes sense. However, since jgit needs to peel these objects also, it doesn't make sense to me that this would be the bottleneck unless git is doing something terribly inefficient here. :(

Except for the fact that repacking objects made it faster, my observations make it look like it's the writing that is actually slow, not the reads. Could there be too many small unbuffered writes, could this write path have missed being optimized (it likely isn't used elsewhere)?

-Martin

Re: Slow git pack-refs --all

[Jeff King]

On Wed, Dec 31, 2025 at 05:48:11AM +0000, Martin Fick wrote:

> >> I think this is from `should_pack_ref`:
> >>
> >>     /* Do not pack broken refs: */
> >>     if (!ref_resolves_to_object(ref->name, refs->base.repo, ref->oid, ref->flags))
> >>       return 0;
> >>
> >> So Git is going to need to verify that the object at least exists.  I
> >> don't know why we would need to _open_ them, however.  Perhaps someone
> >> else has ideas.
> >
> >The packed-refs file stores tag-peeling information. So pack-refs opens
> >the object for any newly written ref via peel_object(), which has to at
> >least read the header to get the type. That call happens via
> >write_with_updates() in packed-backend.c.
> 
> Thanks, this makes sense. However, since jgit needs to peel these
> objects also, it doesn't make sense to me that this would be the
> bottleneck unless git is doing something terribly inefficient here. :(

I'd expect both git and jgit to open each loose object once. I tried
running "git pack-refs --all --prune" under strace on a test repo. It
does seem to open the object once. Then I tried the same with jgit
(though it does not understand --prune), and got the same results.

So...I dunno.

> Except for the fact that repacking objects made it faster, my
> observations make it look like it's the writing that is actually slow,
> not the reads. Could there be too many small unbuffered writes, could
> this write path have missed being optimized (it likely isn't used
> elsewhere)?

All of the packed-refs writes are through fprintf(), which should be
fully buffered. You should be able to confirm with strace (I get
4096-byte writes on my system).

If writing were slow, I'd also expect that to scale with the total
number of refs, not the number of changed refs (since we have to rewrite
the whole file, but only new entries need to be peeled).

-Peff

Re: Slow git pack-refs --all

[Martin Fick]

> From: Jeff King <peff@peff.net>
>  Sent: Friday, January 2, 2026 12:49 AM
> On Wed, Dec 31, 2025 at 05:48:11AM +0000, Martin Fick wrote:
> > Except for the fact that repacking objects made it faster, my

I have now confirmed that repacking does NOT actually make things 
faster. I believe filesystem caching interfered with much of my 
testing. Using echo 3 > /proc/sys/vm/drop_caches has helped to
get more consistent results.

By repacking to get one used, and one cruft pack only, and no loose 
objects, I have confirmed that pack-refs it is still slow. This rules out the 
idea that the loose object, or pack file counts were making things slow.

> > observations make it look like it's the writing that is actually slow,
> > not the reads. Could there be too many small unbuffered writes, could
> > this write path have missed being optimized (it likely isn't used
> > elsewhere)?
>
> All of the packed-refs writes are through fprintf(), which should be
> fully buffered. You should be able to confirm with strace (I get
> 4096-byte writes on my system).

I can confirm that my system is actually printing 8192 bytes at a time.

> If writing were slow, I'd also expect that to scale with the total
> number of refs, not the number of changed refs (since we have to rewrite
> the whole file, but only new entries need to be peeled).

OK, after discovering the strace -r and -T options, I have determined that
the 29K writes were all very fast in themselves. However, most of the
writes seem to follow each other with no other system calls in between.
This explains why it looks like the writes are slow, even though they aren't.

If I tally up the time between the previous system call, and each write(),
it adds up to the bulk of the time (4mins out of 4m15s) that it takes to
pack refs. This tells me that no visible I/O or system calls are the problem,
but rather that the program itself is taking a long time between writes.
I very much doubt that this is heavy CPU time, but rather I am going to 
guess that this is hidden system time spent accessing mmaped memory. 
Could it be really slow reading the packed-refs file? I can see the 
packed-refs file is mmaped() before the writes start, and then 
munmapped after the writes are completed. If I had to guess, that likely
means that the packed-refs file is being read in small increments by the 
kernel via mmap, and that is what is making things very slow over NFS. 
My alternative theory, is that each ref is being looked up via a binary 
search, but I don't think git does this?

-Martin

Re: Slow git pack-refs --all

[Patrick Steinhardt]

Hi Martin,

On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:
> OK, after discovering the strace -r and -T options, I have determined that
> the 29K writes were all very fast in themselves. However, most of the
> writes seem to follow each other with no other system calls in between.
> This explains why it looks like the writes are slow, even though they aren't.
> 
> If I tally up the time between the previous system call, and each write(),
> it adds up to the bulk of the time (4mins out of 4m15s) that it takes to
> pack refs. This tells me that no visible I/O or system calls are the problem,
> but rather that the program itself is taking a long time between writes.
> I very much doubt that this is heavy CPU time, but rather I am going to 
> guess that this is hidden system time spent accessing mmaped memory. 
> Could it be really slow reading the packed-refs file? I can see the 
> packed-refs file is mmaped() before the writes start, and then 
> munmapped after the writes are completed. If I had to guess, that likely
> means that the packed-refs file is being read in small increments by the 
> kernel via mmap, and that is what is making things very slow over NFS. 

I wouldn't be surprised if NFS was the culprit. At GitLab we found it to
be a constant source of issues, which is why we eventually sunsetted the
use of it completely. Do you use any special flags for mounting the NFS
filesystem?

> My alternative theory, is that each ref is being looked up via a binary 
> search, but I don't think git does this?

Did you try using perf(1) to profile the process and generate a flame
graph from it? That should likely make it immediately obvious where Git
is spending all of its time.

Patrick

Re: Slow git pack-refs --all

[Martin Fick]

> From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM
> On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:
> > OK, after discovering the strace -r and -T options, I have determined that
> > the 29K writes were all very fast in themselves. However, most of the
> > writes seem to follow each other with no other system calls in between.
> > This explains why it looks like the writes are slow, even though they aren't.
> >
> > If I tally up the time between the previous system call, and each write(),
> > it adds up to the bulk of the time (4mins out of 4m15s) that it takes to
> > pack refs. This tells me that no visible I/O or system calls are the problem,
> > but rather that the program itself is taking a long time between writes.
> > I very much doubt that this is heavy CPU time, but rather I am going to
> > guess that this is hidden system time spent accessing mmaped memory.
> > Could it be really slow reading the packed-refs file? I can see the
> > packed-refs file is mmaped() before the writes start, and then
> > munmapped after the writes are completed. If I had to guess, that likely
> > means that the packed-refs file is being read in small increments by the
> > kernel via mmap, and that is what is making things very slow over NFS.
> 
> I wouldn't be surprised if NFS was the culprit. At GitLab we found it to
> be a constant source of issues, which is why we eventually sunsetted the
> use of it completely. Do you use any special flags for mounting the NFS
> filesystem?

I am open to alternatives to NFS. Do you know of any NFS alternatives that 
provides instantaneous replication to potentially hundreds of mirrors? I 
have used Gerrit and git-daemon for many years on NFS, and it generally 
has performed very well for us, and it solves many real performance issues 
which I have yet to find a viable alternative able to even come close to
matching. NFS with all it warts it is for us (and likely will be for many) until 
there is a viable enterprise ready alternative with low (zero) replication 
latency and high throughput.

That being said, NFS can cause many issues. In this case, I would say that
something is particularly "broken" here with git, and I believe that it
would be helpful to the git community to be aware of this fairly specific 
broken case which clearly has a lot of room for improvement (as seen
by the fact that jgit, in java, can do essentially the same thing more 
than 10Xs faster). While I have been mostly assuming that this is a 
particularly specific bad case since git daemon generally is fast for most
users, this might actually be something that if improved would greatly 
improve many parts of git (not just this use case).

It would be nice to improve git to not hold the packed-refs.lock so long 
to avoid this blocking behavior on servers. Of course, to be fair, this 
likely only blocks Gerrit servers since Gerrit uses the packed-refs file to 
perform atomic updates for many things, and most other servers use 
loose refs instead. It would be great if git were optimized to avoid any 
unnecessary reads while the lock is held.  In theory, almost all of the 
data that git needs to read here (including tags for peeling) could be 
read before acquiring the lock, and it would only need to double 
check certain reads after it acquires the lock in case things changed. 
That wouldn't make git pack-refs faster, but it would drastically 
reduce the impact of any problematic I/O by not holding the lock for 
almost the entire operation.

> Did you try using perf(1) to profile the process and generate a flame
> graph from it? That should likely make it immediately obvious where Git
> is spending all of its time.

I will pursue this. Unfortunately this might be difficult on this 
particular server.

Thanks for the feedback, and suggestions,

-Martin

Re: Slow git pack-refs --all

[Patrick Steinhardt]

On Tue, Jan 06, 2026 at 11:02:19PM +0000, Martin Fick wrote:
> > From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM
> > On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:
> > > OK, after discovering the strace -r and -T options, I have determined that
> > > the 29K writes were all very fast in themselves. However, most of the
> > > writes seem to follow each other with no other system calls in between.
> > > This explains why it looks like the writes are slow, even though they aren't.
> > >
> > > If I tally up the time between the previous system call, and each write(),
> > > it adds up to the bulk of the time (4mins out of 4m15s) that it takes to
> > > pack refs. This tells me that no visible I/O or system calls are the problem,
> > > but rather that the program itself is taking a long time between writes.
> > > I very much doubt that this is heavy CPU time, but rather I am going to
> > > guess that this is hidden system time spent accessing mmaped memory.
> > > Could it be really slow reading the packed-refs file? I can see the
> > > packed-refs file is mmaped() before the writes start, and then
> > > munmapped after the writes are completed. If I had to guess, that likely
> > > means that the packed-refs file is being read in small increments by the
> > > kernel via mmap, and that is what is making things very slow over NFS.
> > 
> > I wouldn't be surprised if NFS was the culprit. At GitLab we found it to
> > be a constant source of issues, which is why we eventually sunsetted the
> > use of it completely. Do you use any special flags for mounting the NFS
> > filesystem?
> 
> I am open to alternatives to NFS. Do you know of any NFS alternatives that 
> provides instantaneous replication to potentially hundreds of mirrors? I 
> have used Gerrit and git-daemon for many years on NFS, and it generally 
> has performed very well for us, and it solves many real performance issues 
> which I have yet to find a viable alternative able to even come close to
> matching. NFS with all it warts it is for us (and likely will be for many) until 
> there is a viable enterprise ready alternative with low (zero) replication 
> latency and high throughput.

Yeah, agreed, NFS can get you a long way, until you eventually start to
hit some road blocks once you reach a certain scale. Unfortunately
though, there isn't really a ready-made alternative solution that serves
your needs, or at least none that I know of. That's why GitLab
eventually settled on Gitaly Cluster with Praefect handling replication,
and why GitHub has its Spokes architecture that does basically the same
thing.

> That being said, NFS can cause many issues. In this case, I would say that
> something is particularly "broken" here with git, and I believe that it
> would be helpful to the git community to be aware of this fairly specific 
> broken case which clearly has a lot of room for improvement (as seen
> by the fact that jgit, in java, can do essentially the same thing more 
> than 10Xs faster). While I have been mostly assuming that this is a 
> particularly specific bad case since git daemon generally is fast for most
> users, this might actually be something that if improved would greatly 
> improve many parts of git (not just this use case).

Chances are that if we can improve the case for NFS, other filesystems
might benefit, as well. So if this is something that we can improve I
agree that we should. It's too early to tell though, as we don't really
know what the actual root cause is just yet.

> It would be nice to improve git to not hold the packed-refs.lock so long 
> to avoid this blocking behavior on servers. Of course, to be fair, this 
> likely only blocks Gerrit servers since Gerrit uses the packed-refs file to 
> perform atomic updates for many things, and most other servers use 
> loose refs instead. It would be great if git were optimized to avoid any 
> unnecessary reads while the lock is held.  In theory, almost all of the 
> data that git needs to read here (including tags for peeling) could be 
> read before acquiring the lock, and it would only need to double 
> check certain reads after it acquires the lock in case things changed. 
> That wouldn't make git pack-refs faster, but it would drastically 
> reduce the impact of any problematic I/O by not holding the lock for 
> almost the entire operation.

It can probably be improved, true. I think that it's a bit of a wasted
effort, as I'd rather invest the time into improving reftables as a more
future-proof solution. But as you are well aware I'm quite biased here,
and I'd welcome any efforts to also improve the files backend. I am just
unlikely to work on it myself :)

> > Did you try using perf(1) to profile the process and generate a flame
> > graph from it? That should likely make it immediately obvious where Git
> > is spending all of its time.
> 
> I will pursue this. Unfortunately this might be difficult on this 
> particular server.

True, on the server side this can be a bit tricky.

Patrick

Re: Slow git pack-refs --all

[Martin Fick]

> From: Patrick Steinhardt <ps@pks.im> Sent: Wednesday, January 7, 2026 4:42 AM
On Tue, Jan 06, 2026 at 11:02:19PM +0000, Martin Fick wrote:
> > From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM
> > > Did you try using perf(1) to profile the process and generate a flame
> > > graph from it? That should likely make it immediately obvious where Git
> > > is spending all of its time.
> >
> > I will pursue this. Unfortunately this might be difficult on this
> > particular server.
> 
> True, on the server side this can be a bit tricky.

I ran perf, and got a flame graph, I am not sure what the best way to share that
is, but I will try to summarize what looked important:

About one third of the time is in this section:

libc-2.17.so 32.5%
 _memcmp_sse4_1 29.8%
 page_fault 7.23%
 ...

I am not really sure what that is doing?


Another third is doing:

unpack_object_header_buffer 30%
 page_fault 26.9%
 ...
 nfs_read_page 10%

Which could very well be looking at the headers of objects to see if they are 
tags needing to be peeled?


And the remaining third was a bit all over the place with small sections,
the largest two of those sections being:

packed_refs_store_create ~8.7%
 unknown 4.4%
 memchr 4.4%
 page_fault 4.4%

nth_packed_object_offset 7%
 page_fault 3.2%

This was way less informative (to me) then I would have hoped. :( Maybe
this means more to you? 

It does look like a lot of page_faults, likely due to the use of mmap?

-Martin

Re: Slow git pack-refs --all

[Patrick Steinhardt]

On Wed, Jan 07, 2026 at 10:58:36PM +0000, Martin Fick wrote:
> > From: Patrick Steinhardt <ps@pks.im> Sent: Wednesday, January 7, 2026 4:42 AM
> On Tue, Jan 06, 2026 at 11:02:19PM +0000, Martin Fick wrote:
> > > From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM
> > > > Did you try using perf(1) to profile the process and generate a flame
> > > > graph from it? That should likely make it immediately obvious where Git
> > > > is spending all of its time.
> > >
> > > I will pursue this. Unfortunately this might be difficult on this
> > > particular server.
> > 
> > True, on the server side this can be a bit tricky.
> 
> I ran perf, and got a flame graph, I am not sure what the best way to share that
> is, but I will try to summarize what looked important:
> 
> About one third of the time is in this section:
> 
> libc-2.17.so 32.5%
>  _memcmp_sse4_1 29.8%
>  page_fault 7.23%
>  ...
> 
> I am not really sure what that is doing?
> 
> 
> Another third is doing:
> 
> unpack_object_header_buffer 30%
>  page_fault 26.9%
>  ...
>  nfs_read_page 10%
> 
> Which could very well be looking at the headers of objects to see if they are 
> tags needing to be peeled?

Both of these are lacking some information to be able to tell. Are you
by any chance able to share the whole flame graph? That'd make this a
bit easier to figure out.

> And the remaining third was a bit all over the place with small sections,
> the largest two of those sections being:
> 
> packed_refs_store_create ~8.7%
>  unknown 4.4%
>  memchr 4.4%
>  page_fault 4.4%

We spend ~9% of time in `packed_refs_store_create()`? That looks
seriously broken to me, the function shouldn't even do much.

> nth_packed_object_offset 7%
>  page_fault 3.2%
> 
> This was way less informative (to me) then I would have hoped. :( Maybe
> this means more to you? 
> 
> It does look like a lot of page_faults, likely due to the use of mmap?

Certainly looks like the page faults are to blame here overall. It's
still surprising to me it's _that_ slow. Quoting the other mail you sent:

On Wed, Jan 07, 2026 at 05:05:53PM +0000, Martin Fick wrote:
> rw,intr,retrans=10,timeo=600,hard,rsize=32768,wsize=32768,tcp,noacl,_netdev

I know that back when we still supported NFS we recommended to use an
rsize and wsize of 1MB to reduce the round trip times.

Patrick

Re: Slow git pack-refs --all

[Martin Fick]

> From: Patrick Steinhardt <ps@pks.im> Sent: Monday, January 5, 2026 11:53 PM
> On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:
> > OK, after discovering the strace -r and -T options, I have determined that
> > the 29K writes were all very fast in themselves. However, most of the
> > writes seem to follow each other with no other system calls in between.
> > This explains why it looks like the writes are slow, even though they aren't.
> 
> > If I tally up the time between the previous system call, and each write(),
> > it adds up to the bulk of the time (4mins out of 4m15s) that it takes to
> > pack refs. This tells me that no visible I/O or system calls are the problem,
> > but rather that the program itself is taking a long time between writes.
> > I very much doubt that this is heavy CPU time, but rather I am going to
> > guess that this is hidden system time spent accessing mmaped memory.
> > Could it be really slow reading the packed-refs file? I can see the
> > packed-refs file is mmaped() before the writes start, and then
> > munmapped after the writes are completed. If I had to guess, that likely
> > means that the packed-refs file is being read in small increments by the
> > kernel via mmap, and that is what is making things very slow over NFS.
> 
> ... Do you use any special flags for mounting the NFS filesystem?

Oh sorry, I forgot to reply to this last time. We use the following mount flags:

rw,intr,retrans=10,timeo=600,hard,rsize=32768,wsize=32768,tcp,noacl,_netdev

-Martin

Re: Slow git pack-refs --all

[Jeff King]

On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:

> By repacking to get one used, and one cruft pack only, and no loose 
> objects, I have confirmed that pack-refs it is still slow. This rules out the 
> idea that the loose object, or pack file counts were making things slow.

OK, that is interesting. I'd still expect opening the objects to be the
dominating factor, but now the load would be on jumping around the
mmap'd packfile rather than open/read/close calls.

> OK, after discovering the strace -r and -T options, I have determined that
> the 29K writes were all very fast in themselves. However, most of the
> writes seem to follow each other with no other system calls in between.
> This explains why it looks like the writes are slow, even though they aren't.
> 
> If I tally up the time between the previous system call, and each write(),
> it adds up to the bulk of the time (4mins out of 4m15s) that it takes to
> pack refs. This tells me that no visible I/O or system calls are the problem,
> but rather that the program itself is taking a long time between writes.
> I very much doubt that this is heavy CPU time, but rather I am going to 
> guess that this is hidden system time spent accessing mmaped memory.

That would be consistent with reading object data from the packfile.
We'll jump around within the packfile to get that data.

> Could it be really slow reading the packed-refs file? I can see the 
> packed-refs file is mmaped() before the writes start, and then 
> munmapped after the writes are completed. If I had to guess, that likely
> means that the packed-refs file is being read in small increments by the 
> kernel via mmap, and that is what is making things very slow over NFS.

The packed-refs file is mmap'd, but we'll be reading it sequentially. I
guess whether or not there is good read-ahead there may depend on the
NFS implementation.

> My alternative theory, is that each ref is being looked up via a binary 
> search, but I don't think git does this?

Git does binary search within the packed-refs file, but it shouldn't be
doing so here. The write-out phase of packing refs is a straight merge
between two lists: the existing packed-refs entries and the new entries
we are adding.

I'd second Patrick's suggestion to use perf or similar to try to see
where the time is going.

You might also try building Git with NO_MMAP. That might make the I/O
costs more apparent via strace, because they'll be coming via pread().

-Peff

Re: Slow git pack-refs --all

[Martin Fick]

From: Jeff King <peff@peff.net> Sent: Tuesday, January 6, 2026 3:38 AM

> On Mon, Jan 05, 2026 at 11:45:41PM +0000, Martin Fick wrote:
> > By repacking to get one used, and one cruft pack only, and no loose
> > objects, I have confirmed that pack-refs it is still slow. This rules out the
> > idea that the loose object, or pack file counts were making things slow.
> 
> OK, that is interesting. I'd still expect opening the objects to be the
> dominating factor, but now the load would be on jumping around the
> mmap'd packfile rather than open/read/close calls.

I believe I have confirmed this now with more testing...

By first dropping the system caches, and then catting the pack file to
/dev/null, it sped things up to under 20s!

Note that neither catting the idx, nor the packed-refs file helped to 
noticeably speed things up.

> > OK, after discovering the strace -r and -T options, I have determined that
> > the 29K writes were all very fast in themselves. However, most of the
> > writes seem to follow each other with no other system calls in between.
> > This explains why it looks like the writes are slow, even though they aren't.
>
> > If I tally up the time between the previous system call, and each write(),
> > it adds up to the bulk of the time (4mins out of 4m15s) that it takes to
> > pack refs. This tells me that no visible I/O or system calls are the problem,
> > but rather that the program itself is taking a long time between writes.
> > I very much doubt that this is heavy CPU time, but rather I am going to
> > guess that this is hidden system time spent accessing mmaped memory.
> 
> That would be consistent with reading object data from the packfile.
> We'll jump around within the packfile to get that data.

Agreed, but boy is that really bad performance!


> > Could it be really slow reading the packed-refs file? I can see the
> > packed-refs file is mmaped() before the writes start, and then
> > munmapped after the writes are completed. If I had to guess, that likely
> > means that the packed-refs file is being read in small increments by the
> > kernel via mmap, and that is what is making things very slow over NFS.
> 
> The packed-refs file is mmap'd, but we'll be reading it sequentially. I
> guess whether or not there is good read-ahead there may depend on the
> NFS implementation.

Yeah, ruled out now by dropping the system caches, and then catting the 
packed-refs file before running git pack-refs, which did NOT help speed 
things up.


> > My alternative theory, is that each ref is being looked up via a binary
> > search, but I don't think git does this?
> 
> Git does binary search within the packed-refs file, but it shouldn't be
> doing so here. The write-out phase of packing refs is a straight merge
> between two lists: the existing packed-refs entries and the new entries
> we are adding.

Agreed, and I should have ruled this out by realizing that this would likely
not have been affected by the system caches in my earlier tests.


> I'd second Patrick's suggestion to use perf or similar to try to see
> where the time is going.

Noted, thanks.


> You might also try building Git with NO_MMAP. That might make the I/O
> costs more apparent via strace, because they'll be coming via pread().

Agreed, I will try to do this. I think that the jgit results hint that this 
this might even eliminate most of the I/O costs (jgit is not using
MMAP in my tests). It would be nice if this were a runtime config
instead of requiring a rebuild, as some use cases might be better
with, and some without MMAP.

Thanks for all the input,

-Martin

Re: Slow git pack-refs --all

[Martin Fick]

>From: brian m. carlson
> Sent: Thursday, December 25, 2025 4:38 PM
> On 2025-12-25 at 22:13:54, Martin Fick wrote:
>> Although the repository is not in terrible shape before packing refs
>> (~1500 loose objects, 37pack files). Surprisingly, repacking the repo
>> first does speed it up so that packing refs then takes under 20s.
>>
>> This repository is on NFS.

> That's almost certainly part of your performance problem, too.  Loading
> a single pack file and index is going to be way, way faster than making
> lots of network calls to open 37 pack file and 37 index files, plus at
> least stat some loose objects.
>
> I will note that at least some forges always have Git write pack files
> and try to avoid loose objects altogether since that almost always
> improves performance.  You may want to set `receive.unpackLimit` to 1 to
> see if that helps in the general case.

This would not explain why jgit can pack-refs much faster, since it has to deal with these NFS latencies also. In my experience with jgit, we generally don't see performance issues unless packfile counts exceed 300 or so on NFS, definitely not with only 37 of them. It could be that git is doing some things less efficiently here, but I would be pretty surprised if git could not also perform well typically on NFS with only 37 packfiles. I don't think that 200 something object lookups, should ever take 3+mins, even on NFS, and even with way more packfiles than this. To be that slow, it would have to take about 1s per lookup! Something really seems fishy here to me. I have to think that somehow it isn't these reads that are slow, but I can't explain what else it would be?

-Martin