From: "SZEDER Gábor" <szeder.dev@gmail.com>
To: Git Mailing List <git@vger.kernel.org>
Subject: rather slow 'git repack' in 'blob:none' partial clones
Date: Sat, 3 Apr 2021 11:04:25 +0200 [thread overview]
Message-ID: <20210403090412.GH2271@szeder.dev> (raw)
Hi,
here are trace timings of running 'git gc' in a "normal" and in a
'blob:none' partial clone:
$ git clone --bare https://github.com/git/git git-full.git
$ GIT_TRACE_PERFORMANCE=2 /usr/bin/time --format='elapsed: %E max RSS: %Mk' git -C git-full.git/ gc
10:35:24.007277 trace.c:487 performance: 0.001550225 s: git command: /usr/local/libexec/git-core/git pack-refs --all --prune
10:35:24.044641 trace.c:487 performance: 0.035631270 s: git command: /usr/local/libexec/git-core/git reflog expire --all
10:35:24.061070 read-cache.c:2315 performance: 0.000008506 s: read cache ./index
Enumerating objects: 305283, done.
Counting objects: 100% (305283/305283), done.
Delta compression using up to 4 threads
Compressing objects: 100% (75016/75016), done.
Writing objects: 100% (305283/305283), done.
Total 305283 (delta 227928), reused 305283 (delta 227928), pack-reused 0
10:35:32.604546 trace.c:487 performance: 8.555651283 s: git command: /usr/local/libexec/git-core/git pack-objects --local --delta-base-offset objects/pack/.tmp-2946975-pack --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --unpack-unreachable=2.weeks.ago
10:35:32.680597 trace.c:487 performance: 8.633068356 s: git command: /usr/local/libexec/git-core/git repack -d -l -A --unpack-unreachable=2.weeks.ago
10:35:32.683130 trace.c:487 performance: 0.000959377 s: git command: /usr/local/libexec/git-core/git prune --expire 2.weeks.ago
10:35:32.684401 trace.c:487 performance: 0.000180173 s: git command: /usr/local/libexec/git-core/git worktree prune --expire 3.months.ago
10:35:32.685730 trace.c:487 performance: 0.000263898 s: git command: /usr/local/libexec/git-core/git rerere gc
10:35:33.514816 trace.c:487 performance: 9.511597988 s: git command: git -C git-full.git/ gc
elapsed: 0:09.51 max RSS: 358964k
$ git clone --bare --filter=blob:none https://github.com/git/git git-partial.git
$ GIT_TRACE_PERFORMANCE=2 /usr/bin/time --format='elapsed: %E max RSS: %Mk' git -C git-partial.git/ gc
10:35:47.637735 trace.c:487 performance: 0.000872539 s: git command: /usr/local/libexec/git-core/git pack-refs --all --prune
10:35:47.675498 trace.c:487 performance: 0.036246403 s: git command: /usr/local/libexec/git-core/git reflog expire --all
Enumerating objects: 188205, done.
Counting objects: 100% (188205/188205), done.
Delta compression using up to 4 threads
Compressing objects: 100% (66520/66520), done.
Writing objects: 100% (188205/188205), done.
Total 188205 (delta 119967), reused 188205 (delta 119967), pack-reused 0
10:35:50.081709 trace.c:487 performance: 2.402625839 s: git command: /usr/local/libexec/git-core/git pack-objects --local --delta-base-offset objects/pack/.tmp-2946990-pack
10:35:50.100131 read-cache.c:2315 performance: 0.000009979 s: read cache ./index
10:37:04.973541 trace.c:487 performance: 74.885793630 s: git command: /usr/local/libexec/git-core/git pack-objects --local --delta-base-offset objects/pack/.tmp-2946990-pack --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --exclude-promisor-objects --unpack-unreachable=2.weeks.ago
Removing duplicate objects: 100% (256/256), done.
10:37:07.482791 trace.c:487 performance: 79.804973525 s: git command: /usr/local/libexec/git-core/git repack -d -l -A --unpack-unreachable=2.weeks.ago
10:37:07.549333 trace.c:487 performance: 0.008025426 s: git command: /usr/local/libexec/git-core/git prune --expire 2.weeks.ago --exclude-promisor-objects
10:37:07.552499 trace.c:487 performance: 0.000362981 s: git command: /usr/local/libexec/git-core/git worktree prune --expire 3.months.ago
10:37:07.554521 trace.c:487 performance: 0.000273834 s: git command: /usr/local/libexec/git-core/git rerere gc
10:37:10.168233 trace.c:487 performance: 82.533331484 s: git command: git -C git-partial.git/ gc
elapsed: 1:22.54 max RSS: 1891832k
Notice the ~9s vs. 82s runtime and ~350M vs. 1.9G memory consumption
increase. What's going on here?
Also note that that second 'git pack-objects' invocation doesn't show
any progress for ~75s.
FWIW, doing the same in a 'tree:0' partial clone is fast.
next reply other threads:[~2021-04-03 9:04 UTC|newest]
Thread overview: 46+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-03 9:04 SZEDER Gábor [this message]
2021-04-05 1:02 ` rather slow 'git repack' in 'blob:none' partial clones Rafael Silva
2021-04-07 21:17 ` Jeff King
2021-04-08 0:02 ` Jonathan Tan
2021-04-08 0:35 ` Jeff King
2021-04-12 7:09 ` Rafael Silva
2021-04-12 21:36 ` SZEDER Gábor
2021-04-12 21:49 ` Bryan Turner
2021-04-12 23:51 ` Jeff King
2021-04-12 23:47 ` Jeff King
2021-04-13 7:12 ` [PATCH 0/3] low-hanging performance fruit with promisor packs Jeff King
2021-04-13 7:15 ` [PATCH 1/3] is_promisor_object(): free tree buffer after parsing Jeff King
2021-04-13 20:17 ` Junio C Hamano
2021-04-14 5:18 ` Jeff King
2021-04-13 7:16 ` [PATCH 2/3] lookup_unknown_object(): take a repository argument Jeff King
2021-04-13 7:17 ` [PATCH 3/3] revision: avoid parsing with --exclude-promisor-objects Jeff King
2021-04-13 20:22 ` Junio C Hamano
2021-04-13 18:10 ` [PATCH 0/3] low-hanging performance fruit with promisor packs SZEDER Gábor
2021-04-14 17:14 ` Jonathan Tan
2021-04-14 19:22 ` Rafael Silva
2021-04-13 18:05 ` rather slow 'git repack' in 'blob:none' partial clones SZEDER Gábor
2021-04-14 5:14 ` Jeff King
2021-04-11 10:59 ` SZEDER Gábor
2021-04-12 7:53 ` Rafael Silva
2021-04-14 19:14 ` [PATCH 0/2] prevent `repack` to unpack and delete promisor objects Rafael Silva
2021-04-14 19:14 ` [PATCH 1/2] repack: teach --no-prune-packed to skip `git prune-packed` Rafael Silva
2021-04-14 23:50 ` Jonathan Tan
2021-04-18 14:15 ` Rafael Silva
2021-04-14 19:14 ` [PATCH 2/2] repack: avoid loosening promisor pack objects in partial clones Rafael Silva
2021-04-15 1:04 ` Jonathan Tan
2021-04-15 3:51 ` Junio C Hamano
2021-04-15 9:03 ` Jeff King
2021-04-15 9:05 ` Jeff King
2021-04-18 7:12 ` Rafael Silva
2021-04-15 18:06 ` Junio C Hamano
2021-04-18 8:40 ` Rafael Silva
2021-04-14 22:10 ` [PATCH 0/2] prevent `repack` to unpack and delete promisor objects Junio C Hamano
2021-04-15 9:15 ` Jeff King
2021-04-18 8:20 ` Rafael Silva
2021-04-18 13:57 ` [PATCH v2 0/1] " Rafael Silva
2021-04-18 13:57 ` [PATCH v2 1/1] repack: avoid loosening promisor objects in partial clones Rafael Silva
2021-04-19 19:15 ` Jonathan Tan
2021-04-21 18:54 ` Rafael Silva
2021-04-19 23:09 ` Junio C Hamano
2021-04-21 19:25 ` Rafael Silva
2021-04-21 19:32 ` [PATCH v3] " Rafael Silva
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210403090412.GH2271@szeder.dev \
--to=szeder.dev@gmail.com \
--cc=git@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.