Re: Clones with fetch.bundleURI slower than standard, full clone?

["Knop, Ryszard" <ryszard.knop@intel.com>]

On Wed, 2026-07-29 at 00:14 +0000, brian m. carlson wrote:
> On 2026-07-28 at 23:42:03, Knop, Ryszard wrote:
> > Hey all,
> > 
> > I'm working on a Linux kernel-related CI system where we need to
> > perform full clones of the kernel repo in most jobs. Because of some
> > jobs in the pipeline, it usually cannot be a shallow clone :( Since the
> > kernel repo is large and slow to clone, and I don't want to put undue
> > stress on the remote host, I used git bundles, where CI clones the repo
> > over a weekend, packages that as a bundle, then in jobs it gets used
> > like this (weird, but works for <REF> being a branch, tag or a specific
> > commit hash):
> > 
> > git init
> > git remote add origin <REPO-URL>
> > git config set fetch.bundleURI <BUNDLE-URL>
> > git fetch origin <REF>
> > git reset --hard FETCH_HEAD
> > 
> > Cloning a repo this way takes 4-5mins. Unpacking a bundle appears to be
> > super slow. Not even faster than just running a full, normal clone from
> > the remote server, actually (~3-4mins for a single branch).
> 
> You're comparing apples to oranges here.  A clone of a single branch
> includes only that line of history and only those objects, but when you
> use a bundle with multiple refs, Git has to handle all of the objects in
> the bundle's entire pack, not just the ref you've specified.  In order
> to compare adequately, you'd have to compare a bundle containing only
> that one ref with the single-branch clone or a regular clone of the full
> repository with your full bundles.

I thought that due to changes mentioned in this post, Git 2.50 and
newer should just take in all the bundle changes and there should not
be much of a difference during index packs:

https://blog.gitbutler.com/going-down-the-rabbit-hole-of-gits-new-bundle-uri

> 
> > On one of the build VMs, with Git 2.53 (stock Ubuntu 26.04), GIT_TRACE
> > suggests most of the time is spent in some variation of `/usr/lib/git-
> > core/git index-pack --stdin -v --fix-thin '--keep=fetch-pack 39430 on
> > build-server' --check-self-contained-and-connected`, and indeed that
> > process burns 100% of its single thread for most of that time.
> 
> As far as I can tell, the unbundling code just calls index-pack, so it
> should honour pack.threads.  Setting that value to 0 causes this code to
> be executed:
> 
> 		/*
> 		 * Experiments show that going above 20 threads doesn't help,
> 		 * no matter how many cores you have. Below that, we tend to
> 		 * max at half the number of online_cpus(), presumably because
> 		 * half of those are hyperthreads rather than full cores. We'll
> 		 * never reduce the level below "3", though, to match a
> 		 * historical value that nobody complained about.
> 		 */
> 		if (nr_threads < 4)
> 			; /* too few cores to consider capping */
> 		else if (nr_threads < 6)
> 			nr_threads = 3; /* historic cap */
> 		else if (nr_threads < 40)
> 			nr_threads /= 2;
> 		else
> 			nr_threads = 20; /* hard cap */
> 
> So I would expect this to not be single threaded unless Git was compiled
> without pthreads, run on a machine with few cores, or configured to use
> only a single thread.  If you can get threading to work here, I expect
> it will perform better, although I don't have any experience with bundle
> URIs so I can't really say for certain.

Ohhh, this is way better, that halved the checkout time. Thank you! It
seems like the process still has some single-threaded sections during
packing, but it's a problem for another day.

> > Is it expected that doing it this way is so slow? The alternative is to
> > just package and work with the whole bare repo, but bundles appear to
> > be an elegant way of dealing with exactly this scenario.
> 
> To be clear, it is insecure to do anything with an untrusted repo except
> clone or fetch from it, so you will almost certainly not want to
> distribute bare repos, since that will encourage people to use them
> as-is (which is insecure).  Even if these are internal users who can
> trust you, it encourages an anti-pattern which has security problems in
> the general case.

I know it's not ideal, but our reference repos are trusted and the
cache files are used in CI only, so it would probably be fine(tm). With
that config change above I don't have to fall back to this though.

Thanks again, Ryszard