* Re: git add --intent-to-add + git stash "Cannot save the current worktree state"
From: Johannes Schindelin @ 2019-01-09 18:24 UTC (permalink / raw)
To: Anthony Sottile; +Cc: Git Mailing List
In-Reply-To: <CA+dzEBmh4LdN-XFS9y9YKrDMzV_7+QpDzBA2uv9Xi80PgR5NTQ@mail.gmail.com>
Hi Anthony,
On Thu, 3 Jan 2019, Anthony Sottile wrote:
> Minimal reproduction
>
> ```
> git init t
> git -C t commit --allow-empty -m 'initial commit'
> touch t/a
> git -C t add --intent-to-add a
> git -C t stash
> ```
>
> ```
> + git init t
> Initialized empty Git repository in /private/tmp/t/t/.git/
> + git -C t commit --allow-empty -m 'initial commit'
> [master (root-commit) 858132e] initial commit
> + touch t/a
> + git -C t add --intent-to-add a
> + git -C t stash
> error: Entry 'a' not uptodate. Cannot merge.
> Cannot save the current worktree state
> ```
Apparently it is even worse. On Windows, this reportedly resulted in a
segmentation fault:
https://github.com/git-for-windows/git/issues/2006
Could you cherry-pick the fix of
https://github.com/git-for-windows/git/pull/2008 and see whether it fixes
your use case, too?
Ciao,
Johannes
P.S.: Obviously, if this PR fixes the issue, the corresponding change
should be squashed into the appropriate commit in ps/stash-in-c.
^ permalink raw reply
* Re: [PATCH v3 0/2] support for filtering trees and blobs based on depth
From: Jonathan Tan @ 2019-01-09 18:06 UTC (permalink / raw)
To: matvore; +Cc: git, gitster, Jonathan Tan
In-Reply-To: <20190109025914.247473-1-matvore@google.com>
> This applies suggestions from Jonathan Tan and Junio. These are mostly
> stylistic and readability changes, although there is also an added test case
> in t/t6112-rev-list-filters-objects.sh which checks for the scenario when
> filtering which would exclude a blob, but the blob is given on the command
> line.
>
> This has been rebased onto master, while the prior version was based on next.
>
> Thank you,
Thanks, these 2 patches are Reviewed-by: me.
Your approach in the 2nd patch makes more sense, and I checked that both
oidset_insert() and oidset_remove() return 1 when the element in
question was in the set (prior to invocation of the function), so that
works.
^ permalink raw reply
* Re: Make "git log --count" work like "git rev-list"
From: Linus Torvalds @ 2019-01-09 17:44 UTC (permalink / raw)
To: Stefan Beller; +Cc: Junio Hamano C, Git List Mailing
In-Reply-To: <CAGZ79kaOcoVSRgTR+k9ENoDz6FHh27MzpZ9D-MeFArN2vgBK0w@mail.gmail.com>
On Wed, Jan 9, 2019 at 9:21 AM Stefan Beller <sbeller@google.com> wrote:
>
> Sounds reasonable to me to have such functionality,
> as I tend to use
> git log --oneline origin/master..origin/next --no-merges |wc -l
> for such queries, which I always assume to be doing useless
> work as I'd be interested in *only* the count, and not the
> intermediate oneline output, but that is the best output
> that wc works on.
Right. I've been known to do that too, but because I grew up with "git
rev-list", I know about --count and tend to use it.
In fact, I've occasionally used it with "git log" already (before my
patch), and it would silently accept the parameter (because it's
parsed by the generic revision parsing), it just wouldn't work.
> So maybe the --count option would want to suppress
> other output if given (or we'd want to have another option
> for no output)?
It already does so in my patch, exactly because it uses the same logic
as "git rev-list --count" does.
Which is to simply react to the "--count" thing early in
log_tree_commit(), and do the counting and then say "I showed this"
(without showing anything).
So you can do silly things like this:
[torvalds@i7 linux]$ git log --count -10
10
[torvalds@i7 linux]$
and it just works. Nonsensical, but logical.
Linus
^ permalink raw reply
* Re: Regression: submodule worktrees can clobber core.worktree config
From: Stefan Beller @ 2019-01-09 17:42 UTC (permalink / raw)
To: Tomasz Śniatowski, Duy Nguyen; +Cc: git
In-Reply-To: <CAG0vfyQeA3Hm7AsYgYtP4v-Yg0=rKXW0YYfg_emAwEscZha4VA@mail.gmail.com>
On Tue, Jan 8, 2019 at 2:16 PM Tomasz Śniatowski <tsniatowski@vewd.com> wrote:
>
> After upgrading to 2.20.1 I noticed in some submodule+worktree scenarios git
> will break the submodule configuration. Reproducible with:
> git init a && (cd a; touch a; git add a; git commit -ma)
> git init b && (cd b; git submodule add ../a; git commit -mb)
> git -C b worktree add ../b2
> git -C b/a worktree add ../../b2/a
> git -C b status
> git -C b2 submodule update
> git -C b status
>
> The submodule update in the _worktree_ puts an invalid core.worktree value in
> the _original_ repository submodule config (b/.git/modules/a/config), causing
> the last git status to error out with:
> fatal: cannot chdir to '../../../../../../b2/a': No such file or directory
> fatal: 'git status --porcelain=2' failed in submodule a
>
> Looking at the config file itself, the submodule update operation applies the
> following change (the new path is invalid):
> - worktree = ../../../a
> + worktree = ../../../../../../b2/a
>
> This worked fine on 2.19.2 (no config change, no error), and was useful to have
> a worktree with (large) submodules that are also worktrees.
Thanks for reporting the issue!
>
> Bisects down to:
> 74d4731da1 submodule--helper: replace connect-gitdir-workingtree by
> ensure-core-worktree
So this would need to update the worktree config, not the generic config.
We'd need to replace the line
cfg_file = repo_git_path(&subrepo, "config");
in builtin/submodule--helper.c::ensure_core_worktree()
to be a worktree specific call.
Or the other way round we'd want to make repo_git_path to
be worktree specific and introduce repo_common_path for
the main working tree.
Looking at Duys tree,
https://gitlab.com/pclouds/git/commit/94751ada7c32eb6fb2c67dd7723161d1955a5683
is pretty much what we need.
Reverting that topic that introduced this (4d6d6e,
Merge branch 'sb/submodule-update-in-c'), might be possible but
that would conflict with another followup that fixes issues in
that series
(see sb/submodule-unset-core-worktree-when-worktree-is-lost
https://github.com/gitster/git/commits/sb/submodule-unset-core-worktree-when-worktree-is-lost)
so I'd rather just cherry-pick the commit from Duy.
Stefan
^ permalink raw reply
* Re: Make "git log --count" work like "git rev-list"
From: Stefan Beller @ 2019-01-09 17:21 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio Hamano C, Git List Mailing
In-Reply-To: <CAHk-=wg0NUNFjZumgC-9f=kmU3L4T+qOAgXwiDAfPaNtuFfvFg@mail.gmail.com>
On Sat, Jan 5, 2019 at 2:47 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> This is a ridiculous patch. And I understand entirely if nobody else
> cares, because I don't think anybody else has ever even noticed.
>
> It turns out that I still use "git rev-list" outside of some hard-core
> scripting for one silly reason: the linux-next statistics are all
> about non-merge commits, and so to do a rough comparison with
> linux-next, I do
>
> git rev-list --count --no-merges v4.20..
>
> to get an approximate idea of how much I've merged compared to what is
> in linux-next.
>
> (See also
>
> http://neuling.org/linux-next-size.html
>
> for the graphical view of it all, even though it looks a bit odd right
> now because of how linux-next wasn't being updated over the holidats
> and right at the 4.19 release).
>
> Anyway, I've occasionally thought to myself that I should just fix
> "git log" to learn that too, so that I wouldn't have to type out "git
> rev-list". Because "git log" does actually take the "--count"
> argument, it just doesn't honor it.
>
> This is that patch.
Sounds reasonable to me to have such functionality,
as I tend to use
git log --oneline origin/master..origin/next --no-merges |wc -l
for such queries, which I always assume to be doing useless
work as I'd be interested in *only* the count, and not the
intermediate oneline output, but that is the best output
that wc works on.
So maybe the --count option would want to suppress
other output if given (or we'd want to have another option
for no output)?
Instead of printing, do we want to redirect to
rev->diffopt.file, or send it through the diff codes
buffer if we ever anticipate needing this output in
buffered?
A test would be nice, too.
Stefan
^ permalink raw reply
* Re: [PATCH v3 1/3] t5323: test cases for git-pack-redundant
From: SZEDER Gábor @ 2019-01-09 16:47 UTC (permalink / raw)
To: Jiang Xin; +Cc: Sun Chao, Git List, Junio C Hamano, Jiang Xin
In-Reply-To: <20190109125628.GG4673@szeder.dev>
On Wed, Jan 09, 2019 at 01:56:28PM +0100, SZEDER Gábor wrote:
> On Wed, Jan 02, 2019 at 12:34:54PM +0800, Jiang Xin wrote:
> > +cat >expected <<EOF
> > +P1:$P1
> > +P4:$P4
> > +P5:$P5
> > +P6:$P6
> > +EOF
Creating the expected results could be moved into the
test_expect_success block as well.
> > +
> > +test_expect_success 'git pack-redundant --all' '
> > + git pack-redundant --all | \
>
> Don't run a git command (especially the particular command the test
> script focuses on) upstream of a pipe, because it hides the command's
> exit code. Use an intermediate file instead.
>
> > + sed -e "s#^.*/pack-\(.*\)\.\(idx\|pack\)#\1#g" | \
>
> This sed command doesn't seem to work on macOS (on Travis CI), and
> causes the test to fail with:
>
> ++git pack-redundant --all
> ++sed -e 's#^.*/pack-\(.*\)\.\(idx\|pack\)#\1#g'
> ++sort -u
> ++read p
> ++sort
> ++eval echo '${P.git/objects/pack/pack-0cf5cb6afaa1bae36b8e61ca398dbe29a15bc74e.idx}'
> ./test-lib.sh: line 697: ${P.git/objects/pack/pack-0cf5cb6afaa1bae36b8e61ca398dbe29a15bc74e.idx}: bad substitution
> ++test_cmp expected actual
> ++diff -u expected actual
> --- expected 2019-01-09 01:53:45.000000000 +0000
> +++ actual 2019-01-09 01:53:45.000000000 +0000
> @@ -1,4 +0,0 @@
> -P1:24ee080366509364d04a138cd4e168dc4ff33354
> -P4:139d8b0cfe7e8970a8f3533835f90278d88de474
> -P5:23e0f02d822fa4bfe5ee63337ba5632cd7be208e
> -P6:deeb289f1749972f1cd57c3b9f359ece2361f60a
> error: last command exited with $?=1
> not ok 2 - git pack-redundant --all
>
> I'm not sure what's wrong with it, though.
So, it appears that 'sed' in macOS doesn't understand the
'\(idx\|pack\)' part of that regex. Turning that command into
sed -e "s#^.git/objects/pack/pack-\($OID_REGEX\)\..*#\1#" out | \
makes it work even on macOS, but note that those 40 hexdigits are not
actual OIDs but file content checksums, so using $OID_REGEX is not the
right thing to do here (though I'm not sure what is supposed to be
used instead, as $_x40 hardcodes the number of hexdigits).
Alas, the test as a whole still fails with the following on macOS:
++diff -u expected actual
--- expected 2019-01-09 15:54:49.000000000 +0000
+++ actual 2019-01-09 15:54:49.000000000 +0000
@@ -1,4 +1,4 @@
P1:24ee080366509364d04a138cd4e168dc4ff33354
-P4:139d8b0cfe7e8970a8f3533835f90278d88de474
+P3:0cf5cb6afaa1bae36b8e61ca398dbe29a15bc74e
P5:23e0f02d822fa4bfe5ee63337ba5632cd7be208e
-P6:deeb289f1749972f1cd57c3b9f359ece2361f60a
+P7:4ecc1eb138516a26654cd4e3570b322c0820f170
error: last command exited with $?=1
^ permalink raw reply
* Re: [git-users] git checkout file with custom mtime
From: Konstantin Khomoutov @ 2019-01-09 16:20 UTC (permalink / raw)
To: git-users; +Cc: Git List
In-Reply-To: <CAPktr3k6GmYkTE9=0jcs8+4BEeeAnvQa6daxnwcqTwrL91Y-ug@mail.gmail.com>
On Sat, Jan 05, 2019 at 10:44:47PM +0100, Daniel Fanjul wrote:
> I'm on Ubuntu. I do not use LFS. I track mods and saved games of
> Skyrim with git, TESV.exe sorts the saved games only by their mtime. I
> know it is not the most usual use case for git.
>
> I agree with that viewpoint and I like the way git works right now, I
> do not want to change that. Checking out the saved games and then
> fixing the mtime works but forces a lot of unneeded I/O.
You might find this overview of git-annex vs git lfs useful:
https://lwn.net/Articles/774125/
Also see this recent thread on the Git list:
https://public-inbox.org/git/79fd2b4e-243c-a9f5-3485-2954fb0f50ef@aixigo.de/
^ permalink raw reply
* Re: [PATCH v3 7/9] multi-pack-index: prepare 'repack' subcommand
From: SZEDER Gábor @ 2019-01-09 15:56 UTC (permalink / raw)
To: Derrick Stolee via GitGitGadget
Cc: git, sbeller, peff, jrnieder, avarab, Junio C Hamano,
Derrick Stolee
In-Reply-To: <b39f90ad09265614efd466fee5089354a193ae8c.1547047269.git.gitgitgadget@gmail.com>
On Wed, Jan 09, 2019 at 07:21:17AM -0800, Derrick Stolee via GitGitGadget wrote:
> Introduce a 'repack' subcommand to 'git multi-pack-index' that
> takes a '--batch-size' option. The verb will inspect the
s/verb/subcommand/
> diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c
> index 145de3a46c..d87a2235e3 100644
> --- a/builtin/multi-pack-index.c
> +++ b/builtin/multi-pack-index.c
> @@ -40,6 +43,11 @@ int cmd_multi_pack_index(int argc, const char **argv,
> return 1;
> }
>
> + if (!strcmp(argv[0], "repack"))
> + return midx_repack(opts.object_dir, (size_t)opts.batch_size);
> + if (opts.batch_size)
> + die(_("--batch-size option is only for 'repack' verb"));
Likewise.
^ permalink raw reply
* Re: [PATCH v3 6/9] multi-pack-index: implement 'expire' verb
From: SZEDER Gábor @ 2019-01-09 15:54 UTC (permalink / raw)
To: Derrick Stolee via GitGitGadget
Cc: git, sbeller, peff, jrnieder, avarab, Junio C Hamano,
Derrick Stolee
In-Reply-To: <4dbff40e7ad653d9c5c3b19dd63a345a21a73dbd.1547047269.git.gitgitgadget@gmail.com>
On Wed, Jan 09, 2019 at 07:21:16AM -0800, Derrick Stolee via GitGitGadget wrote:
> The 'git multi-pack-index expire' command ...
The subject line could use a s/verb/subcommand/.
^ permalink raw reply
* [PATCH v3 2/9] Docs: rearrange subcommands for multi-pack-index
From: Derrick Stolee via GitGitGadget @ 2019-01-09 15:21 UTC (permalink / raw)
To: git; +Cc: sbeller, peff, jrnieder, avarab, Junio C Hamano, Derrick Stolee
In-Reply-To: <pull.92.v3.git.gitgitgadget@gmail.com>
From: Derrick Stolee <dstolee@microsoft.com>
We will add new subcommands to the multi-pack-index, and that will
make the documentation a bit messier. Clean up the 'verb'
descriptions by renaming the concept to 'subcommand' and removing
the reference to the object directory.
Helped-by: Stefan Beller <sbeller@google.com>
Helped-by: Szeder Gábor <szeder.dev@gmail.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
Documentation/git-multi-pack-index.txt | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt
index f7778a2c85..1af406aca2 100644
--- a/Documentation/git-multi-pack-index.txt
+++ b/Documentation/git-multi-pack-index.txt
@@ -9,7 +9,7 @@ git-multi-pack-index - Write and verify multi-pack-indexes
SYNOPSIS
--------
[verse]
-'git multi-pack-index' [--object-dir=<dir>] <verb>
+'git multi-pack-index' [--object-dir=<dir>] <subcommand>
DESCRIPTION
-----------
@@ -23,13 +23,13 @@ OPTIONS
`<dir>/packs/multi-pack-index` for the current MIDX file, and
`<dir>/packs` for the pack-files to index.
+The following subcommands are available:
+
write::
- When given as the verb, write a new MIDX file to
- `<dir>/packs/multi-pack-index`.
+ Write a new MIDX file.
verify::
- When given as the verb, verify the contents of the MIDX file
- at `<dir>/packs/multi-pack-index`.
+ Verify the contents of the MIDX file.
EXAMPLES
--
gitgitgadget
^ permalink raw reply related
* [PATCH v3 7/9] multi-pack-index: prepare 'repack' subcommand
From: Derrick Stolee via GitGitGadget @ 2019-01-09 15:21 UTC (permalink / raw)
To: git; +Cc: sbeller, peff, jrnieder, avarab, Junio C Hamano, Derrick Stolee
In-Reply-To: <pull.92.v3.git.gitgitgadget@gmail.com>
From: Derrick Stolee <dstolee@microsoft.com>
In an environment where the multi-pack-index is useful, it is due
to many pack-files and an inability to repack the object store
into a single pack-file. However, it is likely that many of these
pack-files are rather small, and could be repacked into a slightly
larger pack-file without too much effort. It may also be important
to ensure the object store is highly available and the repack
operation does not interrupt concurrent git commands.
Introduce a 'repack' subcommand to 'git multi-pack-index' that
takes a '--batch-size' option. The verb will inspect the
multi-pack-index for referenced pack-files whose size is smaller
than the batch size, until collecting a list of pack-files whose
sizes sum to larger than the batch size. Then, a new pack-file
will be created containing the objects from those pack-files that
are referenced by the multi-pack-index. The resulting pack is
likely to actually be smaller than the batch size due to
compression and the fact that there may be objects in the pack-
files that have duplicate copies in other pack-files.
The current change introduces the command-line arguments, and we
add a test that ensures we parse these options properly. Since
we specify a small batch size, we will guarantee that future
implementations do not change the list of pack-files.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
Documentation/git-multi-pack-index.txt | 11 +++++++++++
builtin/multi-pack-index.c | 10 +++++++++-
midx.c | 5 +++++
midx.h | 1 +
t/t5319-multi-pack-index.sh | 11 +++++++++++
5 files changed, 37 insertions(+), 1 deletion(-)
diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt
index 6186c4c936..cc63531cc0 100644
--- a/Documentation/git-multi-pack-index.txt
+++ b/Documentation/git-multi-pack-index.txt
@@ -36,6 +36,17 @@ expire::
have no objects referenced by the MIDX. Rewrite the MIDX file
afterward to remove all references to these pack-files.
+repack::
+ Collect a batch of pack-files whose size are all at most the
+ size given by --batch-size, but whose sizes sum to larger
+ than --batch-size. The batch is selected by greedily adding
+ small pack-files starting with the oldest pack-files that fit
+ the size. Create a new pack-file containing the objects the
+ multi-pack-index indexes into those pack-files, and rewrite
+ the multi-pack-index to contain that pack-file. A later run
+ of 'git multi-pack-index expire' will delete the pack-files
+ that were part of this batch.
+
EXAMPLES
--------
diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c
index 145de3a46c..d87a2235e3 100644
--- a/builtin/multi-pack-index.c
+++ b/builtin/multi-pack-index.c
@@ -5,12 +5,13 @@
#include "midx.h"
static char const * const builtin_multi_pack_index_usage[] = {
- N_("git multi-pack-index [--object-dir=<dir>] (write|verify|expire)"),
+ N_("git multi-pack-index [--object-dir=<dir>] (write|verify|expire|repack --batch-size=<size>)"),
NULL
};
static struct opts_multi_pack_index {
const char *object_dir;
+ unsigned long batch_size;
} opts;
int cmd_multi_pack_index(int argc, const char **argv,
@@ -19,6 +20,8 @@ int cmd_multi_pack_index(int argc, const char **argv,
static struct option builtin_multi_pack_index_options[] = {
OPT_FILENAME(0, "object-dir", &opts.object_dir,
N_("object directory containing set of packfile and pack-index pairs")),
+ OPT_MAGNITUDE(0, "batch-size", &opts.batch_size,
+ N_("during repack, collect pack-files of smaller size into a batch that is larger than this size")),
OPT_END(),
};
@@ -40,6 +43,11 @@ int cmd_multi_pack_index(int argc, const char **argv,
return 1;
}
+ if (!strcmp(argv[0], "repack"))
+ return midx_repack(opts.object_dir, (size_t)opts.batch_size);
+ if (opts.batch_size)
+ die(_("--batch-size option is only for 'repack' verb"));
+
if (!strcmp(argv[0], "write"))
return write_midx_file(opts.object_dir);
if (!strcmp(argv[0], "verify"))
diff --git a/midx.c b/midx.c
index 6ccbec3f19..30ff4430ab 100644
--- a/midx.c
+++ b/midx.c
@@ -1106,3 +1106,8 @@ int expire_midx_packs(const char *object_dir)
string_list_clear(&packs_to_drop, 0);
return result;
}
+
+int midx_repack(const char *object_dir, size_t batch_size)
+{
+ return 0;
+}
diff --git a/midx.h b/midx.h
index e3a2b740b5..394a21ee96 100644
--- a/midx.h
+++ b/midx.h
@@ -50,6 +50,7 @@ int write_midx_file(const char *object_dir);
void clear_midx_file(struct repository *r);
int verify_midx_file(const char *object_dir);
int expire_midx_packs(const char *object_dir);
+int midx_repack(const char *object_dir, size_t batch_size);
void close_midx(struct multi_pack_index *m);
diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh
index f55a60a89c..d64767600a 100755
--- a/t/t5319-multi-pack-index.sh
+++ b/t/t5319-multi-pack-index.sh
@@ -413,4 +413,15 @@ test_expect_success 'expire removes unreferenced packs' '
)
'
+test_expect_success 'repack with minimum size does not alter existing packs' '
+ (
+ cd dup &&
+ ls .git/objects/pack >expect &&
+ MINSIZE=$(ls -l .git/objects/pack/*pack | awk "{print \$5;}" | sort -n | head -n 1) &&
+ git multi-pack-index repack --batch-size=$MINSIZE &&
+ ls .git/objects/pack >actual &&
+ test_cmp expect actual
+ )
+'
+
test_done
--
gitgitgadget
^ permalink raw reply related
* [PATCH v3 9/9] multi-pack-index: test expire while adding packs
From: Derrick Stolee via GitGitGadget @ 2019-01-09 15:21 UTC (permalink / raw)
To: git; +Cc: sbeller, peff, jrnieder, avarab, Junio C Hamano, Derrick Stolee
In-Reply-To: <pull.92.v3.git.gitgitgadget@gmail.com>
From: Derrick Stolee <dstolee@microsoft.com>
During development of the multi-pack-index expire subcommand, a
version went out that improperly computed the pack order if a new
pack was introduced while other packs were being removed. Part of
the subtlety of the bug involved the new pack being placed before
other packs that already existed in the multi-pack-index.
Add a test to t5319-multi-pack-index.sh that catches this issue.
The test adds new packs that cause another pack to be expired, and
creates new packs that are lexicographically sorted before and
after the existing packs.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
t/t5319-multi-pack-index.sh | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)
diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh
index 99afb5ec51..1213549d25 100755
--- a/t/t5319-multi-pack-index.sh
+++ b/t/t5319-multi-pack-index.sh
@@ -449,4 +449,36 @@ test_expect_success 'expire removes repacked packs' '
)
'
+test_expect_success 'expire works when adding new packs' '
+ (
+ cd dup &&
+ git pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&
+ refs/heads/A
+ ^refs/heads/B
+ EOF
+ git pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&
+ refs/heads/B
+ ^refs/heads/C
+ EOF
+ git pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&
+ refs/heads/C
+ ^refs/heads/D
+ EOF
+ git multi-pack-index write &&
+ git pack-objects --revs .git/objects/pack/a-pack <<-EOF &&
+ refs/heads/D
+ ^refs/heads/E
+ EOF
+ git multi-pack-index write &&
+ git pack-objects --revs .git/objects/pack/z-pack <<-EOF &&
+ refs/heads/E
+ EOF
+ git multi-pack-index expire &&
+ ls .git/objects/pack/ | grep idx >expect &&
+ test-tool read-midx .git/objects | grep idx >actual &&
+ test_cmp expect actual &&
+ git multi-pack-index verify
+ )
+'
+
test_done
--
gitgitgadget
^ permalink raw reply related
* [PATCH v3 8/9] midx: implement midx_repack()
From: Derrick Stolee via GitGitGadget @ 2019-01-09 15:21 UTC (permalink / raw)
To: git; +Cc: sbeller, peff, jrnieder, avarab, Junio C Hamano, Derrick Stolee
In-Reply-To: <pull.92.v3.git.gitgitgadget@gmail.com>
From: Derrick Stolee <dstolee@microsoft.com>
To repack using a multi-pack-index, first sort all pack-files by
their modified time. Second, walk those pack-files from oldest
to newest, adding the packs to a list if they are smaller than the
given pack-size. Finally, collect the objects from the multi-pack-
index that are in those packs and send them to 'git pack-objects'.
While first designing a 'git multi-pack-index repack' operation, I
started by collecting the batches based on the size of the objects
instead of the size of the pack-files. This allows repacking a
large pack-file that has very few referencd objects. However, this
came at a significant cost of parsing pack-files instead of simply
reading the multi-pack-index and getting the file information for
the pack-files. This object-size idea could be a direction for
future expansion in this area.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
midx.c | 109 +++++++++++++++++++++++++++++++++++-
t/t5319-multi-pack-index.sh | 25 +++++++++
2 files changed, 133 insertions(+), 1 deletion(-)
diff --git a/midx.c b/midx.c
index 30ff4430ab..f0f16690e6 100644
--- a/midx.c
+++ b/midx.c
@@ -8,6 +8,7 @@
#include "sha1-lookup.h"
#include "midx.h"
#include "progress.h"
+#include "run-command.h"
#define MIDX_SIGNATURE 0x4d494458 /* "MIDX" */
#define MIDX_VERSION 1
@@ -1107,7 +1108,113 @@ int expire_midx_packs(const char *object_dir)
return result;
}
-int midx_repack(const char *object_dir, size_t batch_size)
+struct time_and_id {
+ timestamp_t mtime;
+ uint32_t pack_int_id;
+};
+
+static int compare_by_mtime(const void *a_, const void *b_)
{
+ const struct time_and_id *a, *b;
+
+ a = (const struct time_and_id *)a_;
+ b = (const struct time_and_id *)b_;
+
+ if (a->mtime < b->mtime)
+ return -1;
+ if (a->mtime > b->mtime)
+ return 1;
return 0;
}
+
+int midx_repack(const char *object_dir, size_t batch_size)
+{
+ int result = 0;
+ uint32_t i, packs_to_repack;
+ size_t total_size;
+ struct time_and_id *pack_ti;
+ unsigned char *include_pack;
+ struct child_process cmd = CHILD_PROCESS_INIT;
+ struct strbuf base_name = STRBUF_INIT;
+ struct multi_pack_index *m = load_multi_pack_index(object_dir, 1);
+
+ if (!m)
+ return 0;
+
+ include_pack = xcalloc(m->num_packs, sizeof(unsigned char));
+ pack_ti = xcalloc(m->num_packs, sizeof(struct time_and_id));
+
+ for (i = 0; i < m->num_packs; i++) {
+ pack_ti[i].pack_int_id = i;
+
+ if (prepare_midx_pack(m, i))
+ continue;
+
+ pack_ti[i].mtime = m->packs[i]->mtime;
+ }
+ QSORT(pack_ti, m->num_packs, compare_by_mtime);
+
+ total_size = 0;
+ packs_to_repack = 0;
+ for (i = 0; total_size < batch_size && i < m->num_packs; i++) {
+ int pack_int_id = pack_ti[i].pack_int_id;
+ struct packed_git *p = m->packs[pack_int_id];
+
+ if (!p)
+ continue;
+ if (p->pack_size >= batch_size)
+ continue;
+
+ packs_to_repack++;
+ total_size += p->pack_size;
+ include_pack[pack_int_id] = 1;
+ }
+
+ if (total_size < batch_size || packs_to_repack < 2)
+ goto cleanup;
+
+ argv_array_push(&cmd.args, "pack-objects");
+
+ strbuf_addstr(&base_name, object_dir);
+ strbuf_addstr(&base_name, "/pack/pack");
+ argv_array_push(&cmd.args, base_name.buf);
+ strbuf_release(&base_name);
+
+ cmd.git_cmd = 1;
+ cmd.in = cmd.out = -1;
+
+ if (start_command(&cmd)) {
+ error(_("could not start pack-objects"));
+ result = 1;
+ goto cleanup;
+ }
+
+ for (i = 0; i < m->num_objects; i++) {
+ struct object_id oid;
+ uint32_t pack_int_id = nth_midxed_pack_int_id(m, i);
+
+ if (!include_pack[pack_int_id])
+ continue;
+
+ nth_midxed_object_oid(&oid, m, i);
+ xwrite(cmd.in, oid_to_hex(&oid), the_hash_algo->hexsz);
+ xwrite(cmd.in, "\n", 1);
+ }
+ close(cmd.in);
+
+ if (finish_command(&cmd)) {
+ error(_("could not finish pack-objects"));
+ result = 1;
+ goto cleanup;
+ }
+
+ result = write_midx_internal(object_dir, m, NULL);
+ m = NULL;
+
+cleanup:
+ if (m)
+ close_midx(m);
+ free(include_pack);
+ free(pack_ti);
+ return result;
+}
diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh
index d64767600a..99afb5ec51 100755
--- a/t/t5319-multi-pack-index.sh
+++ b/t/t5319-multi-pack-index.sh
@@ -424,4 +424,29 @@ test_expect_success 'repack with minimum size does not alter existing packs' '
)
'
+test_expect_success 'repack creates a new pack' '
+ (
+ cd dup &&
+ SECOND_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk "{print \$5;}" | sort -n | head -n 2 | tail -n 1) &&
+ BATCH_SIZE=$(($SECOND_SMALLEST_SIZE + 1)) &&
+ git multi-pack-index repack --batch-size=$BATCH_SIZE &&
+ ls .git/objects/pack/*idx >idx-list &&
+ test_line_count = 5 idx-list &&
+ test-tool read-midx .git/objects | grep idx >midx-list &&
+ test_line_count = 5 midx-list
+ )
+'
+
+test_expect_success 'expire removes repacked packs' '
+ (
+ cd dup &&
+ ls -S .git/objects/pack/*pack | head -n 3 >expect &&
+ git multi-pack-index expire &&
+ ls -S .git/objects/pack/*pack >actual &&
+ test_cmp expect actual &&
+ test-tool read-midx .git/objects | grep idx >midx-list &&
+ test_line_count = 3 midx-list
+ )
+'
+
test_done
--
gitgitgadget
^ permalink raw reply related
* [PATCH v3 5/9] midx: refactor permutation logic and pack sorting
From: Derrick Stolee via GitGitGadget @ 2019-01-09 15:21 UTC (permalink / raw)
To: git; +Cc: sbeller, peff, jrnieder, avarab, Junio C Hamano, Derrick Stolee
In-Reply-To: <pull.92.v3.git.gitgitgadget@gmail.com>
From: Derrick Stolee <dstolee@microsoft.com>
In anticipation of the expire subcommand, refactor the way we sort
the packfiles by name. This will greatly simplify our approach to
dropping expired packs from the list.
First, create 'struct pack_info' to replace 'struct pack_pair'.
This struct contains the necessary information about a pack,
including its name, a pointer to its packfile struct (if not
already in the multi-pack-index), and the original pack-int-id.
Second, track the pack information using an array of pack_info
structs in the pack_list struct. This simplifies the logic around
the multiple arrays we were tracking in that struct.
Finally, update get_sorted_entries() to not permute the pack-int-id
and instead supply the permutation to write_midx_object_offsets().
This requires sorting the packs after get_sorted_entries().
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
midx.c | 150 ++++++++++++++++++++++++---------------------------------
1 file changed, 63 insertions(+), 87 deletions(-)
diff --git a/midx.c b/midx.c
index f087bbbe82..7ae5275c25 100644
--- a/midx.c
+++ b/midx.c
@@ -377,12 +377,23 @@ static size_t write_midx_header(struct hashfile *f,
return MIDX_HEADER_SIZE;
}
+struct pack_info {
+ uint32_t orig_pack_int_id;
+ char *pack_name;
+ struct packed_git *p;
+};
+
+static int pack_info_compare(const void *_a, const void *_b)
+{
+ struct pack_info *a = (struct pack_info *)_a;
+ struct pack_info *b = (struct pack_info *)_b;
+ return strcmp(a->pack_name, b->pack_name);
+}
+
struct pack_list {
- struct packed_git **list;
- char **names;
+ struct pack_info *info;
uint32_t nr;
- uint32_t alloc_list;
- uint32_t alloc_names;
+ uint32_t alloc;
struct multi_pack_index *m;
};
@@ -395,66 +406,32 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,
if (packs->m && midx_contains_pack(packs->m, file_name))
return;
- ALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list);
- ALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names);
+ ALLOC_GROW(packs->info, packs->nr + 1, packs->alloc);
- packs->list[packs->nr] = add_packed_git(full_path,
- full_path_len,
- 0);
+ packs->info[packs->nr].p = add_packed_git(full_path,
+ full_path_len,
+ 0);
- if (!packs->list[packs->nr]) {
+ if (!packs->info[packs->nr].p) {
warning(_("failed to add packfile '%s'"),
full_path);
return;
}
- if (open_pack_index(packs->list[packs->nr])) {
+ if (open_pack_index(packs->info[packs->nr].p)) {
warning(_("failed to open pack-index '%s'"),
full_path);
- close_pack(packs->list[packs->nr]);
- FREE_AND_NULL(packs->list[packs->nr]);
+ close_pack(packs->info[packs->nr].p);
+ FREE_AND_NULL(packs->info[packs->nr].p);
return;
}
- packs->names[packs->nr] = xstrdup(file_name);
+ packs->info[packs->nr].pack_name = xstrdup(file_name);
+ packs->info[packs->nr].orig_pack_int_id = packs->nr;
packs->nr++;
}
}
-struct pack_pair {
- uint32_t pack_int_id;
- char *pack_name;
-};
-
-static int pack_pair_compare(const void *_a, const void *_b)
-{
- struct pack_pair *a = (struct pack_pair *)_a;
- struct pack_pair *b = (struct pack_pair *)_b;
- return strcmp(a->pack_name, b->pack_name);
-}
-
-static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *perm)
-{
- uint32_t i;
- struct pack_pair *pairs;
-
- ALLOC_ARRAY(pairs, nr_packs);
-
- for (i = 0; i < nr_packs; i++) {
- pairs[i].pack_int_id = i;
- pairs[i].pack_name = pack_names[i];
- }
-
- QSORT(pairs, nr_packs, pack_pair_compare);
-
- for (i = 0; i < nr_packs; i++) {
- pack_names[i] = pairs[i].pack_name;
- perm[pairs[i].pack_int_id] = i;
- }
-
- free(pairs);
-}
-
struct pack_midx_entry {
struct object_id oid;
uint32_t pack_int_id;
@@ -480,7 +457,6 @@ static int midx_oid_compare(const void *_a, const void *_b)
}
static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,
- uint32_t *pack_perm,
struct pack_midx_entry *e,
uint32_t pos)
{
@@ -488,7 +464,7 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,
return 1;
nth_midxed_object_oid(&e->oid, m, pos);
- e->pack_int_id = pack_perm[nth_midxed_pack_int_id(m, pos)];
+ e->pack_int_id = nth_midxed_pack_int_id(m, pos);
e->offset = nth_midxed_offset(m, pos);
/* consider objects in midx to be from "old" packs */
@@ -522,8 +498,7 @@ static void fill_pack_entry(uint32_t pack_int_id,
* of a packfile containing the object).
*/
static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,
- struct packed_git **p,
- uint32_t *perm,
+ struct pack_info *info,
uint32_t nr_packs,
uint32_t *nr_objects)
{
@@ -534,7 +509,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,
uint32_t start_pack = m ? m->num_packs : 0;
for (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++)
- total_objects += p[cur_pack]->num_objects;
+ total_objects += info[cur_pack].p->num_objects;
/*
* As we de-duplicate by fanout value, we expect the fanout
@@ -559,7 +534,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,
for (cur_object = start; cur_object < end; cur_object++) {
ALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);
- nth_midxed_pack_midx_entry(m, perm,
+ nth_midxed_pack_midx_entry(m,
&entries_by_fanout[nr_fanout],
cur_object);
nr_fanout++;
@@ -570,12 +545,12 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,
uint32_t start = 0, end;
if (cur_fanout)
- start = get_pack_fanout(p[cur_pack], cur_fanout - 1);
- end = get_pack_fanout(p[cur_pack], cur_fanout);
+ start = get_pack_fanout(info[cur_pack].p, cur_fanout - 1);
+ end = get_pack_fanout(info[cur_pack].p, cur_fanout);
for (cur_object = start; cur_object < end; cur_object++) {
ALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout);
- fill_pack_entry(perm[cur_pack], p[cur_pack], cur_object, &entries_by_fanout[nr_fanout]);
+ fill_pack_entry(cur_pack, info[cur_pack].p, cur_object, &entries_by_fanout[nr_fanout]);
nr_fanout++;
}
}
@@ -604,7 +579,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m,
}
static size_t write_midx_pack_names(struct hashfile *f,
- char **pack_names,
+ struct pack_info *info,
uint32_t num_packs)
{
uint32_t i;
@@ -612,14 +587,14 @@ static size_t write_midx_pack_names(struct hashfile *f,
size_t written = 0;
for (i = 0; i < num_packs; i++) {
- size_t writelen = strlen(pack_names[i]) + 1;
+ size_t writelen = strlen(info[i].pack_name) + 1;
- if (i && strcmp(pack_names[i], pack_names[i - 1]) <= 0)
+ if (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0)
BUG("incorrect pack-file order: %s before %s",
- pack_names[i - 1],
- pack_names[i]);
+ info[i - 1].pack_name,
+ info[i].pack_name);
- hashwrite(f, pack_names[i], writelen);
+ hashwrite(f, info[i].pack_name, writelen);
written += writelen;
}
@@ -690,6 +665,7 @@ static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len,
}
static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed,
+ uint32_t *perm,
struct pack_midx_entry *objects, uint32_t nr_objects)
{
struct pack_midx_entry *list = objects;
@@ -699,7 +675,7 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee
for (i = 0; i < nr_objects; i++) {
struct pack_midx_entry *obj = list++;
- hashwrite_be32(f, obj->pack_int_id);
+ hashwrite_be32(f, perm[obj->pack_int_id]);
if (large_offset_needed && obj->offset >> 31)
hashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++);
@@ -772,20 +748,17 @@ int write_midx_file(const char *object_dir)
packs.m = load_multi_pack_index(object_dir, 1);
packs.nr = 0;
- packs.alloc_list = packs.m ? packs.m->num_packs : 16;
- packs.alloc_names = packs.alloc_list;
- packs.list = NULL;
- packs.names = NULL;
- ALLOC_ARRAY(packs.list, packs.alloc_list);
- ALLOC_ARRAY(packs.names, packs.alloc_names);
+ packs.alloc = packs.m ? packs.m->num_packs : 16;
+ packs.info = NULL;
+ ALLOC_ARRAY(packs.info, packs.alloc);
if (packs.m) {
for (i = 0; i < packs.m->num_packs; i++) {
- ALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list);
- ALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names);
+ ALLOC_GROW(packs.info, packs.nr + 1, packs.alloc);
- packs.list[packs.nr] = NULL;
- packs.names[packs.nr] = xstrdup(packs.m->pack_names[i]);
+ packs.info[packs.nr].orig_pack_int_id = i;
+ packs.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]);
+ packs.info[packs.nr].p = NULL;
packs.nr++;
}
}
@@ -795,10 +768,7 @@ int write_midx_file(const char *object_dir)
if (packs.m && packs.nr == packs.m->num_packs)
goto cleanup;
- ALLOC_ARRAY(pack_perm, packs.nr);
- sort_packs_by_name(packs.names, packs.nr, pack_perm);
-
- entries = get_sorted_entries(packs.m, packs.list, pack_perm, packs.nr, &nr_entries);
+ entries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);
for (i = 0; i < nr_entries; i++) {
if (entries[i].offset > 0x7fffffff)
@@ -807,8 +777,15 @@ int write_midx_file(const char *object_dir)
large_offsets_needed = 1;
}
+ QSORT(packs.info, packs.nr, pack_info_compare);
+
+ ALLOC_ARRAY(pack_perm, packs.nr);
+ for (i = 0; i < packs.nr; i++) {
+ pack_perm[packs.info[i].orig_pack_int_id] = i;
+ }
+
for (i = 0; i < packs.nr; i++)
- pack_name_concat_len += strlen(packs.names[i]) + 1;
+ pack_name_concat_len += strlen(packs.info[i].pack_name) + 1;
if (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)
pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -
@@ -879,7 +856,7 @@ int write_midx_file(const char *object_dir)
switch (chunk_ids[i]) {
case MIDX_CHUNKID_PACKNAMES:
- written += write_midx_pack_names(f, packs.names, packs.nr);
+ written += write_midx_pack_names(f, packs.info, packs.nr);
break;
case MIDX_CHUNKID_OIDFANOUT:
@@ -891,7 +868,7 @@ int write_midx_file(const char *object_dir)
break;
case MIDX_CHUNKID_OBJECTOFFSETS:
- written += write_midx_object_offsets(f, large_offsets_needed, entries, nr_entries);
+ written += write_midx_object_offsets(f, large_offsets_needed, pack_perm, entries, nr_entries);
break;
case MIDX_CHUNKID_LARGEOFFSETS:
@@ -914,15 +891,14 @@ int write_midx_file(const char *object_dir)
cleanup:
for (i = 0; i < packs.nr; i++) {
- if (packs.list[i]) {
- close_pack(packs.list[i]);
- free(packs.list[i]);
+ if (packs.info[i].p) {
+ close_pack(packs.info[i].p);
+ free(packs.info[i].p);
}
- free(packs.names[i]);
+ free(packs.info[i].pack_name);
}
- free(packs.list);
- free(packs.names);
+ free(packs.info);
free(entries);
free(pack_perm);
free(midx_name);
--
gitgitgadget
^ permalink raw reply related
* [PATCH v3 6/9] multi-pack-index: implement 'expire' verb
From: Derrick Stolee via GitGitGadget @ 2019-01-09 15:21 UTC (permalink / raw)
To: git; +Cc: sbeller, peff, jrnieder, avarab, Junio C Hamano, Derrick Stolee
In-Reply-To: <pull.92.v3.git.gitgitgadget@gmail.com>
From: Derrick Stolee <dstolee@microsoft.com>
The 'git multi-pack-index expire' command looks at the existing
mult-pack-index, counts the number of objects referenced in each
pack-file, deletes the pack-fils with no referenced objects, and
rewrites the multi-pack-index to no longer reference those packs.
Refactor the write_midx_file() method to call write_midx_internal()
which now takes an existing 'struct multi_pack_index' and a list
of pack-files to drop (as specified by the names of their pack-
indexes). As we write the new multi-pack-index, we drop those
file names from the list of known pack-files.
The expire_midx_packs() method removes the unreferenced pack-files
after carefully closing the packs to avoid open handles.
Test that a new pack-file that covers the contents of two other
pack-files leads to those pack-files being deleted during the
expire command. Be sure to read the multi-pack-index to ensure
it no longer references those packs.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
midx.c | 120 +++++++++++++++++++++++++++++++++---
t/t5319-multi-pack-index.sh | 18 ++++++
2 files changed, 128 insertions(+), 10 deletions(-)
diff --git a/midx.c b/midx.c
index 7ae5275c25..6ccbec3f19 100644
--- a/midx.c
+++ b/midx.c
@@ -33,6 +33,8 @@
#define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))
#define MIDX_LARGE_OFFSET_NEEDED 0x80000000
+#define PACK_EXPIRED UINT_MAX
+
static char *get_midx_filename(const char *object_dir)
{
return xstrfmt("%s/pack/multi-pack-index", object_dir);
@@ -381,6 +383,7 @@ struct pack_info {
uint32_t orig_pack_int_id;
char *pack_name;
struct packed_git *p;
+ unsigned expired : 1;
};
static int pack_info_compare(const void *_a, const void *_b)
@@ -428,6 +431,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,
packs->info[packs->nr].pack_name = xstrdup(file_name);
packs->info[packs->nr].orig_pack_int_id = packs->nr;
+ packs->info[packs->nr].expired = 0;
packs->nr++;
}
}
@@ -587,13 +591,17 @@ static size_t write_midx_pack_names(struct hashfile *f,
size_t written = 0;
for (i = 0; i < num_packs; i++) {
- size_t writelen = strlen(info[i].pack_name) + 1;
+ size_t writelen;
+
+ if (info[i].expired)
+ continue;
if (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0)
BUG("incorrect pack-file order: %s before %s",
info[i - 1].pack_name,
info[i].pack_name);
+ writelen = strlen(info[i].pack_name) + 1;
hashwrite(f, info[i].pack_name, writelen);
written += writelen;
}
@@ -675,6 +683,11 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee
for (i = 0; i < nr_objects; i++) {
struct pack_midx_entry *obj = list++;
+ if (perm[obj->pack_int_id] == PACK_EXPIRED)
+ BUG("object %s is in an expired pack with int-id %d",
+ oid_to_hex(&obj->oid),
+ obj->pack_int_id);
+
hashwrite_be32(f, perm[obj->pack_int_id]);
if (large_offset_needed && obj->offset >> 31)
@@ -721,7 +734,8 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off
return written;
}
-int write_midx_file(const char *object_dir)
+static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,
+ struct string_list *packs_to_drop)
{
unsigned char cur_chunk, num_chunks = 0;
char *midx_name;
@@ -737,6 +751,8 @@ int write_midx_file(const char *object_dir)
struct pack_midx_entry *entries = NULL;
int large_offsets_needed = 0;
int pack_name_concat_len = 0;
+ int dropped_packs = 0;
+ int result = 0;
midx_name = get_midx_filename(object_dir);
if (safe_create_leading_directories(midx_name)) {
@@ -745,7 +761,10 @@ int write_midx_file(const char *object_dir)
midx_name);
}
- packs.m = load_multi_pack_index(object_dir, 1);
+ if (m)
+ packs.m = m;
+ else
+ packs.m = load_multi_pack_index(object_dir, 1);
packs.nr = 0;
packs.alloc = packs.m ? packs.m->num_packs : 16;
@@ -759,13 +778,14 @@ int write_midx_file(const char *object_dir)
packs.info[packs.nr].orig_pack_int_id = i;
packs.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]);
packs.info[packs.nr].p = NULL;
+ packs.info[packs.nr].expired = 0;
packs.nr++;
}
}
for_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);
- if (packs.m && packs.nr == packs.m->num_packs)
+ if (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop)
goto cleanup;
entries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);
@@ -779,13 +799,48 @@ int write_midx_file(const char *object_dir)
QSORT(packs.info, packs.nr, pack_info_compare);
+ if (packs_to_drop && packs_to_drop->nr) {
+ int drop_index = 0;
+ int missing_drops = 0;
+
+ for (i = 0; i < packs.nr && drop_index < packs_to_drop->nr; i++) {
+ int cmp = strcmp(packs.info[i].pack_name,
+ packs_to_drop->items[drop_index].string);
+
+ if (!cmp) {
+ drop_index++;
+ packs.info[i].expired = 1;
+ } else if (cmp > 0) {
+ error(_("did not see pack-file %s to drop"),
+ packs_to_drop->items[drop_index].string);
+ drop_index++;
+ missing_drops++;
+ i--;
+ } else {
+ packs.info[i].expired = 0;
+ }
+ }
+
+ if (missing_drops) {
+ result = 1;
+ goto cleanup;
+ }
+ }
+
ALLOC_ARRAY(pack_perm, packs.nr);
for (i = 0; i < packs.nr; i++) {
- pack_perm[packs.info[i].orig_pack_int_id] = i;
+ if (packs.info[i].expired) {
+ dropped_packs++;
+ pack_perm[packs.info[i].orig_pack_int_id] = PACK_EXPIRED;
+ } else {
+ pack_perm[packs.info[i].orig_pack_int_id] = i - dropped_packs;
+ }
}
- for (i = 0; i < packs.nr; i++)
- pack_name_concat_len += strlen(packs.info[i].pack_name) + 1;
+ for (i = 0; i < packs.nr; i++) {
+ if (!packs.info[i].expired)
+ pack_name_concat_len += strlen(packs.info[i].pack_name) + 1;
+ }
if (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)
pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -
@@ -801,7 +856,7 @@ int write_midx_file(const char *object_dir)
cur_chunk = 0;
num_chunks = large_offsets_needed ? 5 : 4;
- written = write_midx_header(f, num_chunks, packs.nr);
+ written = write_midx_header(f, num_chunks, packs.nr - dropped_packs);
chunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;
chunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;
@@ -902,7 +957,12 @@ int write_midx_file(const char *object_dir)
free(entries);
free(pack_perm);
free(midx_name);
- return 0;
+ return result;
+}
+
+int write_midx_file(const char *object_dir)
+{
+ return write_midx_internal(object_dir, NULL, NULL);
}
void clear_midx_file(struct repository *r)
@@ -1004,5 +1064,45 @@ int verify_midx_file(const char *object_dir)
int expire_midx_packs(const char *object_dir)
{
- return 0;
+ uint32_t i, *count, result = 0;
+ struct string_list packs_to_drop = STRING_LIST_INIT_DUP;
+ struct multi_pack_index *m = load_multi_pack_index(object_dir, 1);
+
+ if (!m)
+ return 0;
+
+ count = xcalloc(m->num_packs, sizeof(uint32_t));
+ for (i = 0; i < m->num_objects; i++) {
+ int pack_int_id = nth_midxed_pack_int_id(m, i);
+ count[pack_int_id]++;
+ }
+
+ for (i = 0; i < m->num_packs; i++) {
+ char *pack_name;
+
+ if (count[i])
+ continue;
+
+ if (prepare_midx_pack(m, i))
+ continue;
+
+ if (m->packs[i]->pack_keep)
+ continue;
+
+ pack_name = xstrdup(m->packs[i]->pack_name);
+ close_pack(m->packs[i]);
+ FREE_AND_NULL(m->packs[i]);
+
+ string_list_insert(&packs_to_drop, m->pack_names[i]);
+ unlink_pack_path(pack_name, 0);
+ free(pack_name);
+ }
+
+ free(count);
+
+ if (packs_to_drop.nr)
+ result = write_midx_internal(object_dir, m, &packs_to_drop);
+
+ string_list_clear(&packs_to_drop, 0);
+ return result;
}
diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh
index 948effc1ee..f55a60a89c 100755
--- a/t/t5319-multi-pack-index.sh
+++ b/t/t5319-multi-pack-index.sh
@@ -395,4 +395,22 @@ test_expect_success 'expire does not remove any packs' '
)
'
+test_expect_success 'expire removes unreferenced packs' '
+ (
+ cd dup &&
+ git pack-objects --revs .git/objects/pack/pack-combined <<-EOF &&
+ refs/heads/A
+ ^refs/heads/C
+ EOF
+ git multi-pack-index write &&
+ ls .git/objects/pack | grep -v -e pack-[AB] >expect &&
+ git multi-pack-index expire &&
+ ls .git/objects/pack >actual &&
+ test_cmp expect actual &&
+ ls .git/objects/pack/ | grep idx >expect-idx &&
+ test-tool read-midx .git/objects | grep idx >actual-midx &&
+ test_cmp expect-idx actual-midx
+ )
+'
+
test_done
--
gitgitgadget
^ permalink raw reply related
* [PATCH v3 4/9] midx: simplify computation of pack name lengths
From: Derrick Stolee via GitGitGadget @ 2019-01-09 15:21 UTC (permalink / raw)
To: git; +Cc: sbeller, peff, jrnieder, avarab, Junio C Hamano, Derrick Stolee
In-Reply-To: <pull.92.v3.git.gitgitgadget@gmail.com>
From: Derrick Stolee <dstolee@microsoft.com>
Before writing the multi-pack-index, we compute the length of the
pack-index names concatenated together. This forms the data in the
pack name chunk, and we precompute it to compute chunk offsets.
The value is also modified to fit alignment needs.
Previously, this computation was coupled with adding packs from
the existing multi-pack-index and the remaining packs in the object
dir not already covered by the multi-pack-index.
In anticipation of this becoming more complicated with the 'expire'
command, simplify the computation by centralizing it to a single
loop before writing the file.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
midx.c | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/midx.c b/midx.c
index bb825ef816..f087bbbe82 100644
--- a/midx.c
+++ b/midx.c
@@ -383,7 +383,6 @@ struct pack_list {
uint32_t nr;
uint32_t alloc_list;
uint32_t alloc_names;
- size_t pack_name_concat_len;
struct multi_pack_index *m;
};
@@ -418,7 +417,6 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,
}
packs->names[packs->nr] = xstrdup(file_name);
- packs->pack_name_concat_len += strlen(file_name) + 1;
packs->nr++;
}
}
@@ -762,6 +760,7 @@ int write_midx_file(const char *object_dir)
uint32_t nr_entries, num_large_offsets = 0;
struct pack_midx_entry *entries = NULL;
int large_offsets_needed = 0;
+ int pack_name_concat_len = 0;
midx_name = get_midx_filename(object_dir);
if (safe_create_leading_directories(midx_name)) {
@@ -777,7 +776,6 @@ int write_midx_file(const char *object_dir)
packs.alloc_names = packs.alloc_list;
packs.list = NULL;
packs.names = NULL;
- packs.pack_name_concat_len = 0;
ALLOC_ARRAY(packs.list, packs.alloc_list);
ALLOC_ARRAY(packs.names, packs.alloc_names);
@@ -788,7 +786,6 @@ int write_midx_file(const char *object_dir)
packs.list[packs.nr] = NULL;
packs.names[packs.nr] = xstrdup(packs.m->pack_names[i]);
- packs.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1;
packs.nr++;
}
}
@@ -798,10 +795,6 @@ int write_midx_file(const char *object_dir)
if (packs.m && packs.nr == packs.m->num_packs)
goto cleanup;
- if (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)
- packs.pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -
- (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);
-
ALLOC_ARRAY(pack_perm, packs.nr);
sort_packs_by_name(packs.names, packs.nr, pack_perm);
@@ -814,6 +807,13 @@ int write_midx_file(const char *object_dir)
large_offsets_needed = 1;
}
+ for (i = 0; i < packs.nr; i++)
+ pack_name_concat_len += strlen(packs.names[i]) + 1;
+
+ if (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)
+ pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -
+ (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);
+
hold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR);
f = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf);
FREE_AND_NULL(midx_name);
@@ -831,7 +831,7 @@ int write_midx_file(const char *object_dir)
cur_chunk++;
chunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT;
- chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len;
+ chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len;
cur_chunk++;
chunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP;
--
gitgitgadget
^ permalink raw reply related
* [PATCH v3 3/9] multi-pack-index: prepare for 'expire' subcommand
From: Derrick Stolee via GitGitGadget @ 2019-01-09 15:21 UTC (permalink / raw)
To: git; +Cc: sbeller, peff, jrnieder, avarab, Junio C Hamano, Derrick Stolee
In-Reply-To: <pull.92.v3.git.gitgitgadget@gmail.com>
From: Derrick Stolee <dstolee@microsoft.com>
The multi-pack-index tracks objects in a collection of pack-files.
Only one copy of each object is indexed, using the modified time
of the pack-files to determine tie-breakers. It is possible to
have a pack-file with no referenced objects because all objects
have a duplicate in a newer pack-file.
Introduce a new 'expire' subcommand to the multi-pack-index builtin.
This subcommand will delete these unused pack-files and rewrite the
multi-pack-index to no longer refer to those files. More details
about the specifics will follow as the method is implemented.
Add a test that verifies the 'expire' subcommand is correctly wired,
but will still be valid when the verb is implemented. Specifically,
create a set of packs that should all have referenced objects and
should not be removed during an 'expire' operation.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
Documentation/git-multi-pack-index.txt | 5 +++
builtin/multi-pack-index.c | 4 ++-
midx.c | 5 +++
midx.h | 1 +
t/t5319-multi-pack-index.sh | 47 ++++++++++++++++++++++++++
5 files changed, 61 insertions(+), 1 deletion(-)
diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt
index 1af406aca2..6186c4c936 100644
--- a/Documentation/git-multi-pack-index.txt
+++ b/Documentation/git-multi-pack-index.txt
@@ -31,6 +31,11 @@ write::
verify::
Verify the contents of the MIDX file.
+expire::
+ Delete the pack-files that are tracked by the MIDX file, but
+ have no objects referenced by the MIDX. Rewrite the MIDX file
+ afterward to remove all references to these pack-files.
+
EXAMPLES
--------
diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c
index fca70f8e4f..145de3a46c 100644
--- a/builtin/multi-pack-index.c
+++ b/builtin/multi-pack-index.c
@@ -5,7 +5,7 @@
#include "midx.h"
static char const * const builtin_multi_pack_index_usage[] = {
- N_("git multi-pack-index [--object-dir=<dir>] (write|verify)"),
+ N_("git multi-pack-index [--object-dir=<dir>] (write|verify|expire)"),
NULL
};
@@ -44,6 +44,8 @@ int cmd_multi_pack_index(int argc, const char **argv,
return write_midx_file(opts.object_dir);
if (!strcmp(argv[0], "verify"))
return verify_midx_file(opts.object_dir);
+ if (!strcmp(argv[0], "expire"))
+ return expire_midx_packs(opts.object_dir);
die(_("unrecognized verb: %s"), argv[0]);
}
diff --git a/midx.c b/midx.c
index 730ff84dff..bb825ef816 100644
--- a/midx.c
+++ b/midx.c
@@ -1025,3 +1025,8 @@ int verify_midx_file(const char *object_dir)
return verify_midx_error;
}
+
+int expire_midx_packs(const char *object_dir)
+{
+ return 0;
+}
diff --git a/midx.h b/midx.h
index 774f652530..e3a2b740b5 100644
--- a/midx.h
+++ b/midx.h
@@ -49,6 +49,7 @@ int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, i
int write_midx_file(const char *object_dir);
void clear_midx_file(struct repository *r);
int verify_midx_file(const char *object_dir);
+int expire_midx_packs(const char *object_dir);
void close_midx(struct multi_pack_index *m);
diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh
index 70926b5bc0..948effc1ee 100755
--- a/t/t5319-multi-pack-index.sh
+++ b/t/t5319-multi-pack-index.sh
@@ -348,4 +348,51 @@ test_expect_success 'verify incorrect 64-bit offset' '
"incorrect object offset"
'
+test_expect_success 'setup expire tests' '
+ mkdir dup &&
+ (
+ cd dup &&
+ git init &&
+ for i in $(test_seq 1 20)
+ do
+ test_commit $i
+ done &&
+ git branch A HEAD &&
+ git branch B HEAD~8 &&
+ git branch C HEAD~13 &&
+ git branch D HEAD~16 &&
+ git branch E HEAD~18 &&
+ git pack-objects --revs .git/objects/pack/pack-E <<-EOF &&
+ refs/heads/E
+ EOF
+ git pack-objects --revs .git/objects/pack/pack-D <<-EOF &&
+ refs/heads/D
+ ^refs/heads/E
+ EOF
+ git pack-objects --revs .git/objects/pack/pack-C <<-EOF &&
+ refs/heads/C
+ ^refs/heads/D
+ EOF
+ git pack-objects --revs .git/objects/pack/pack-B <<-EOF &&
+ refs/heads/B
+ ^refs/heads/C
+ EOF
+ git pack-objects --revs .git/objects/pack/pack-A <<-EOF &&
+ refs/heads/A
+ ^refs/heads/B
+ EOF
+ git multi-pack-index write
+ )
+'
+
+test_expect_success 'expire does not remove any packs' '
+ (
+ cd dup &&
+ ls .git/objects/pack >expect &&
+ git multi-pack-index expire &&
+ ls .git/objects/pack >actual &&
+ test_cmp expect actual
+ )
+'
+
test_done
--
gitgitgadget
^ permalink raw reply related
* [PATCH v3 1/9] repack: refactor pack deletion for future use
From: Derrick Stolee via GitGitGadget @ 2019-01-09 15:21 UTC (permalink / raw)
To: git; +Cc: sbeller, peff, jrnieder, avarab, Junio C Hamano, Derrick Stolee
In-Reply-To: <pull.92.v3.git.gitgitgadget@gmail.com>
From: Derrick Stolee <dstolee@microsoft.com>
The repack builtin deletes redundant pack-files and their
associated .idx, .promisor, .bitmap, and .keep files. We will want
to re-use this logic in the future for other types of repack, so
pull the logic into 'unlink_pack_path()' in packfile.c.
The 'ignore_keep' parameter is enabled for the use in repack, but
will be important for a future caller.
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
---
builtin/repack.c | 14 ++------------
packfile.c | 28 ++++++++++++++++++++++++++++
packfile.h | 7 +++++++
3 files changed, 37 insertions(+), 12 deletions(-)
diff --git a/builtin/repack.c b/builtin/repack.c
index 45583683ee..3d445b34b4 100644
--- a/builtin/repack.c
+++ b/builtin/repack.c
@@ -129,19 +129,9 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list,
static void remove_redundant_pack(const char *dir_name, const char *base_name)
{
- const char *exts[] = {".pack", ".idx", ".keep", ".bitmap", ".promisor"};
- int i;
struct strbuf buf = STRBUF_INIT;
- size_t plen;
-
- strbuf_addf(&buf, "%s/%s", dir_name, base_name);
- plen = buf.len;
-
- for (i = 0; i < ARRAY_SIZE(exts); i++) {
- strbuf_setlen(&buf, plen);
- strbuf_addstr(&buf, exts[i]);
- unlink(buf.buf);
- }
+ strbuf_addf(&buf, "%s/%s.pack", dir_name, base_name);
+ unlink_pack_path(buf.buf, 1);
strbuf_release(&buf);
}
diff --git a/packfile.c b/packfile.c
index d1e6683ffe..bacecb4d0d 100644
--- a/packfile.c
+++ b/packfile.c
@@ -352,6 +352,34 @@ void close_all_packs(struct raw_object_store *o)
}
}
+void unlink_pack_path(const char *pack_name, int force_delete)
+{
+ static const char *exts[] = {".pack", ".idx", ".keep", ".bitmap", ".promisor"};
+ int i;
+ struct strbuf buf = STRBUF_INIT;
+ size_t plen;
+
+ strbuf_addstr(&buf, pack_name);
+ strip_suffix_mem(buf.buf, &buf.len, ".pack");
+ plen = buf.len;
+
+ if (!force_delete) {
+ strbuf_addstr(&buf, ".keep");
+ if (!access(buf.buf, F_OK)) {
+ strbuf_release(&buf);
+ return;
+ }
+ }
+
+ for (i = 0; i < ARRAY_SIZE(exts); i++) {
+ strbuf_setlen(&buf, plen);
+ strbuf_addstr(&buf, exts[i]);
+ unlink(buf.buf);
+ }
+
+ strbuf_release(&buf);
+}
+
/*
* The LRU pack is the one with the oldest MRU window, preferring packs
* with no used windows, or the oldest mtime if it has no windows allocated.
diff --git a/packfile.h b/packfile.h
index 6c4037605d..5b7bcdb1dd 100644
--- a/packfile.h
+++ b/packfile.h
@@ -86,6 +86,13 @@ extern void unuse_pack(struct pack_window **);
extern void clear_delta_base_cache(void);
extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
+/*
+ * Unlink the .pack and associated extension files.
+ * Does not unlink if 'force_delete' is false and the pack-file is
+ * marked as ".keep".
+ */
+extern void unlink_pack_path(const char *pack_name, int force_delete);
+
/*
* Make sure that a pointer access into an mmap'd index file is within bounds,
* and can provide at least 8 bytes of data.
--
gitgitgadget
^ permalink raw reply related
* [PATCH v3 0/9] Create 'expire' and 'repack' verbs for git-multi-pack-index
From: Derrick Stolee via GitGitGadget @ 2019-01-09 15:21 UTC (permalink / raw)
To: git; +Cc: sbeller, peff, jrnieder, avarab, Junio C Hamano
In-Reply-To: <pull.92.v2.git.gitgitgadget@gmail.com>
The multi-pack-index provides a fast way to find an object among a large
list of pack-files. It stores a single pack-reference for each object id, so
duplicate objects are ignored. Among a list of pack-files storing the same
object, the most-recently modified one is used.
Create new subcommands for the multi-pack-index builtin.
* 'git multi-pack-index expire': If we have a pack-file indexed by the
multi-pack-index, but all objects in that pack are duplicated in
more-recently modified packs, then delete that pack (and any others like
it). Delete the reference to that pack in the multi-pack-index.
* 'git multi-pack-index repack --batch-size=': Starting from the oldest
pack-files covered by the multi-pack-index, find those whose on-disk size
is below the batch size until we have a collection of packs whose sizes
add up to the batch size. Create a new pack containing all objects that
the multi-pack-index references to those packs.
This allows us to create a new pattern for repacking objects: run 'repack'.
After enough time has passed that all Git commands that started before the
last 'repack' are finished, run 'expire' again. This approach has some
advantages over the existing "repack everything" model:
1. Incremental. We can repack a small batch of objects at a time, instead
of repacking all reachable objects. We can also limit ourselves to the
objects that do not appear in newer pack-files.
2. Highly Available. By adding a new pack-file (and not deleting the old
pack-files) we do not interrupt concurrent Git commands, and do not
suffer performance degradation. By expiring only pack-files that have no
referenced objects, we know that Git commands that are doing normal
object lookups* will not be interrupted.
3. Note: if someone concurrently runs a Git command that uses
get_all_packs(), then that command could try to read the pack-files and
pack-indexes that we are deleting during an expire command. Such
commands are usually related to object maintenance (i.e. fsck, gc,
pack-objects) or are related to less-often-used features (i.e.
fast-import, http-backend, server-info).
We plan to use this approach in VFS for Git to do background maintenance of
the "shared object cache" which is a Git alternate directory filled with
packfiles containing commits and trees. We currently download pack-files on
an hourly basis to keep up-to-date with the central server. The cache
servers supply packs on an hourly and daily basis, so most of the hourly
packs become useless after a new daily pack is downloaded. The 'expire'
command would clear out most of those packs, but many will still remain with
fewer than 100 objects remaining. The 'repack' command (with a batch size of
1-3gb, probably) can condense the remaining packs in commands that run for
1-3 min at a time. Since the daily packs range from 100-250mb, we will also
combine and condense those packs.
Updates in V2:
* Added a method, unlink_pack_path() to remove packfiles, but with the
additional check for a .keep file. This borrows logic from
builtin/repack.c.
* Modified documentation and commit messages to replace 'verb' with
'subcommand'. Simplified the documentation. (I left 'verbs' in the title
of the cover letter for consistency.)
Updates in V3:
* There was a bug in the expire logic when simultaneously removing packs
and adding uncovered packs, specifically around the pack permutation.
This was hard to see during review because I was using the 'pack_perm'
array for multiple purposes. First, I was reducing its length, and then I
was adding to it and resorting. In V3, I significantly overhauled the
logic here, which required some extra commits before implementing
'expire'. The final commit includes a test that would cover this case.
Thanks, -Stolee
Derrick Stolee (9):
repack: refactor pack deletion for future use
Docs: rearrange subcommands for multi-pack-index
multi-pack-index: prepare for 'expire' subcommand
midx: simplify computation of pack name lengths
midx: refactor permutation logic and pack sorting
multi-pack-index: implement 'expire' verb
multi-pack-index: prepare 'repack' subcommand
midx: implement midx_repack()
multi-pack-index: test expire while adding packs
Documentation/git-multi-pack-index.txt | 26 +-
builtin/multi-pack-index.c | 12 +-
builtin/repack.c | 14 +-
midx.c | 393 ++++++++++++++++++-------
midx.h | 2 +
packfile.c | 28 ++
packfile.h | 7 +
t/t5319-multi-pack-index.sh | 133 +++++++++
8 files changed, 497 insertions(+), 118 deletions(-)
base-commit: 26aa9fc81d4c7f6c3b456a29da0b7ec72e5c6595
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-92%2Fderrickstolee%2Fmidx-expire%2Fupstream-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-92/derrickstolee/midx-expire/upstream-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/92
Range-diff vs v2:
1: a697df120c = 1: 62b393b816 repack: refactor pack deletion for future use
2: 55df6b20ff = 2: 7886785904 Docs: rearrange subcommands for multi-pack-index
3: 2529afe89e = 3: f06382b4ae multi-pack-index: prepare for 'expire' subcommand
4: 0c29a242fe < -: ---------- midx: refactor permutation logic
-: ---------- > 4: 2a763990ae midx: simplify computation of pack name lengths
-: ---------- > 5: a0d4cc6cb3 midx: refactor permutation logic and pack sorting
5: 1c4af93f5e ! 6: 4dbff40e7a multi-pack-index: implement 'expire' verb
@@ -26,6 +26,62 @@
diff --git a/midx.c b/midx.c
--- a/midx.c
+++ b/midx.c
+@@
+ #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t))
+ #define MIDX_LARGE_OFFSET_NEEDED 0x80000000
+
++#define PACK_EXPIRED UINT_MAX
++
+ static char *get_midx_filename(const char *object_dir)
+ {
+ return xstrfmt("%s/pack/multi-pack-index", object_dir);
+@@
+ uint32_t orig_pack_int_id;
+ char *pack_name;
+ struct packed_git *p;
++ unsigned expired : 1;
+ };
+
+ static int pack_info_compare(const void *_a, const void *_b)
+@@
+
+ packs->info[packs->nr].pack_name = xstrdup(file_name);
+ packs->info[packs->nr].orig_pack_int_id = packs->nr;
++ packs->info[packs->nr].expired = 0;
+ packs->nr++;
+ }
+ }
+@@
+ size_t written = 0;
+
+ for (i = 0; i < num_packs; i++) {
+- size_t writelen = strlen(info[i].pack_name) + 1;
++ size_t writelen;
++
++ if (info[i].expired)
++ continue;
+
+ if (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0)
+ BUG("incorrect pack-file order: %s before %s",
+ info[i - 1].pack_name,
+ info[i].pack_name);
+
++ writelen = strlen(info[i].pack_name) + 1;
+ hashwrite(f, info[i].pack_name, writelen);
+ written += writelen;
+ }
+@@
+ for (i = 0; i < nr_objects; i++) {
+ struct pack_midx_entry *obj = list++;
+
++ if (perm[obj->pack_int_id] == PACK_EXPIRED)
++ BUG("object %s is in an expired pack with int-id %d",
++ oid_to_hex(&obj->oid),
++ obj->pack_int_id);
++
+ hashwrite_be32(f, perm[obj->pack_int_id]);
+
+ if (large_offset_needed && obj->offset >> 31)
@@
return written;
}
@@ -37,9 +93,10 @@
unsigned char cur_chunk, num_chunks = 0;
char *midx_name;
@@
- uint32_t nr_entries, num_large_offsets = 0;
struct pack_midx_entry *entries = NULL;
int large_offsets_needed = 0;
+ int pack_name_concat_len = 0;
++ int dropped_packs = 0;
+ int result = 0;
midx_name = get_midx_filename(object_dir);
@@ -55,49 +112,87 @@
+ packs.m = load_multi_pack_index(object_dir, 1);
packs.nr = 0;
- packs.alloc_list = packs.m ? packs.m->num_packs : 16;
+ packs.alloc = packs.m ? packs.m->num_packs : 16;
@@
- ALLOC_ARRAY(packs.perm, packs.alloc_perm);
-
- if (packs.m) {
-+ int drop_index = 0, missing_drops = 0;
- for (i = 0; i < packs.m->num_packs; i++) {
-+ if (packs_to_drop && drop_index < packs_to_drop->nr) {
-+ int cmp = strcmp(packs.m->pack_names[i],
-+ packs_to_drop->items[drop_index].string);
-+
-+ if (!cmp) {
-+ drop_index++;
-+ continue;
-+ } else if (cmp > 0) {
-+ error(_("did not see pack-file %s to drop"),
-+ packs_to_drop->items[drop_index].string);
-+ drop_index++;
-+ i--;
-+ missing_drops++;
-+ continue;
-+ }
-+ }
-+
- ALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list);
- ALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names);
- ALLOC_GROW(packs.perm, packs.nr + 1, packs.alloc_perm);
-@@
- packs.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1;
+ packs.info[packs.nr].orig_pack_int_id = i;
+ packs.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]);
+ packs.info[packs.nr].p = NULL;
++ packs.info[packs.nr].expired = 0;
packs.nr++;
}
+ }
+
+ for_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);
+
+- if (packs.m && packs.nr == packs.m->num_packs)
++ if (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop)
+ goto cleanup;
+
+ entries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries);
+@@
+
+ QSORT(packs.info, packs.nr, pack_info_compare);
+
++ if (packs_to_drop && packs_to_drop->nr) {
++ int drop_index = 0;
++ int missing_drops = 0;
++
++ for (i = 0; i < packs.nr && drop_index < packs_to_drop->nr; i++) {
++ int cmp = strcmp(packs.info[i].pack_name,
++ packs_to_drop->items[drop_index].string);
++
++ if (!cmp) {
++ drop_index++;
++ packs.info[i].expired = 1;
++ } else if (cmp > 0) {
++ error(_("did not see pack-file %s to drop"),
++ packs_to_drop->items[drop_index].string);
++ drop_index++;
++ missing_drops++;
++ i--;
++ } else {
++ packs.info[i].expired = 0;
++ }
++ }
+
-+ if (packs_to_drop && (drop_index < packs_to_drop->nr || missing_drops)) {
-+ error(_("did not see all pack-files to drop"));
++ if (missing_drops) {
+ result = 1;
+ goto cleanup;
++ }
++ }
++
+ ALLOC_ARRAY(pack_perm, packs.nr);
+ for (i = 0; i < packs.nr; i++) {
+- pack_perm[packs.info[i].orig_pack_int_id] = i;
++ if (packs.info[i].expired) {
++ dropped_packs++;
++ pack_perm[packs.info[i].orig_pack_int_id] = PACK_EXPIRED;
++ } else {
++ pack_perm[packs.info[i].orig_pack_int_id] = i - dropped_packs;
+ }
}
- for_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs);
+- for (i = 0; i < packs.nr; i++)
+- pack_name_concat_len += strlen(packs.info[i].pack_name) + 1;
++ for (i = 0; i < packs.nr; i++) {
++ if (!packs.info[i].expired)
++ pack_name_concat_len += strlen(packs.info[i].pack_name) + 1;
++ }
+
+ if (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT)
+ pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -
+@@
+ cur_chunk = 0;
+ num_chunks = large_offsets_needed ? 5 : 4;
+
+- written = write_midx_header(f, num_chunks, packs.nr);
++ written = write_midx_header(f, num_chunks, packs.nr - dropped_packs);
+
+ chunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES;
+ chunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH;
@@
- free(packs.perm);
free(entries);
+ free(pack_perm);
free(midx_name);
- return 0;
+ return result;
@@ -175,7 +270,10 @@
+ ls .git/objects/pack | grep -v -e pack-[AB] >expect &&
+ git multi-pack-index expire &&
+ ls .git/objects/pack >actual &&
-+ test_cmp expect actual
++ test_cmp expect actual &&
++ ls .git/objects/pack/ | grep idx >expect-idx &&
++ test-tool read-midx .git/objects | grep idx >actual-midx &&
++ test_cmp expect-idx actual-midx
+ )
+'
+
6: af08e21c97 = 7: b39f90ad09 multi-pack-index: prepare 'repack' subcommand
7: bef7aa007c = 8: a4c2d5a8e1 midx: implement midx_repack()
-: ---------- > 9: b97fb35ba9 multi-pack-index: test expire while adding packs
--
gitgitgadget
^ permalink raw reply
* Re: git rebase: retain original head?
From: Markus Wiederkehr @ 2019-01-09 15:08 UTC (permalink / raw)
To: Git mailing list
In-Reply-To: <nycvar.QRO.7.76.6.1901091501320.41@tvgsbejvaqbjf.bet>
On Wed, Jan 9, 2019 at 3:05 PM Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
>
> Having said that, it is an unintended regression in the built-in rebase.
> Markus, could you come up with a minimal test case, preferably in the form
> of a patch to t/t3415-rebase-autosquash.sh?
I don't think I'm familiar enough with the test code to be able to
provide a good patch but the following code should illustrate the
regression.
git init testrepo
cd testrepo
echo 1 > file && git add file && git commit -m "initial"
echo 1 >> file && git commit -am "commit 1"
# rev_commit_1=$(git rev-parse HEAD)
echo 1 >> file && git commit -am "fixup! $(git rev-parse --short HEAD)"
echo 1 >> file && git commit -am "commit 3"
rev_orig_head=$(git rev-parse HEAD)
GIT_EDITOR=: git rebase --autosquash -i HEAD~3
test $(git rev-parse ORIG_HEAD) = $rev_orig_head
In older versions of git this test succeeded, i.e. ORIG_HEAD pointed
to the previous original head, $rev_orig_head. In git version 2.20.1
ORIG_HEAD now points to the commit that got fixuped instead,
$rev_commit_1.
In previous versions ORIG_HEAD only pointed somewhere else if "git
reset" was invoked manually during the rebase operation. I'm not sure
if this is desirable, maybe ORIG_HEAD should always point to the
previous head after rebase completes, no matter what operations were
run in between. What do you think?
Markus
^ permalink raw reply
* Re: git rebase: retain original head?
From: Johannes Schindelin @ 2019-01-09 14:05 UTC (permalink / raw)
To: Jacob Keller; +Cc: Markus Wiederkehr, Git mailing list
In-Reply-To: <CA+P7+xq++4W32JT9WcasXn=Oj9W-U1eteFgpLZn8GVqeO0foog@mail.gmail.com>
Hi Jake,
On Tue, 8 Jan 2019, Jacob Keller wrote:
> On Tue, Jan 8, 2019 at 12:47 PM Markus Wiederkehr
> <markus.wiederkehr@gmail.com> wrote:
> >
> > On Tue, Jan 8, 2019 at 6:43 PM Andreas Schwab <schwab@linux-m68k.org> wrote:
> > >
> > > On Jan 08 2019, Markus Wiederkehr <markus.wiederkehr@gmail.com> wrote:
> > >
> > > > Would it be possible to retain this information?
> > >
> > > You could use the reflog of the current branch, where it is the second
> > > entry.
> >
> > It is not, depending on what happens in the rebase it could be any
> > entry. That's why I always have to search for the right one, which is
> > tedious.
> >
> > Example:
> >
> > $ git rebase -i root-tag
> > ... (apply one fixup)
> > $ git reflog
> > 906caf1c (HEAD -> master) HEAD@{0}: rebase -i (finish): returning to
> > refs/heads/master
> > 4906caf1c (HEAD -> master) HEAD@{1}: rebase -i (pick): qux
> > 85dab37b4 HEAD@{2}: rebase -i (pick): baz
> > 7de7420d2 HEAD@{3}: rebase -i (fixup): bar
> > 9bc0461c0 HEAD@{4}: rebase -i (start): checkout root-tag
> > a150b73ca HEAD@{5}: commit: foo
> >
> > Here I have to use HEAD@{5}.
> >
>
> He meant using:
>
> $ git reflog master
Indeed. That's why terse answers are often worse than no answers at all:
more often than not, they leave things unclearer than before, in the worst
case they are misleading, as was the very terse first reply. Just don't.
Too short is not beautiful.
> which only shows changes directly to the master ref, not to HEAD.
> During a git rebase operations don't modify the master ref until after
> the rebase finishes. This, of course, assumes you're on a branch.
>
> If so, you should even be able to use master@{1} and be safe in
> assuming it always refers to what the branch was before the rebase
> started.
There are more tricks up Git's sleeves: @{1} is a shorthand for the
previous state of the current branch (if any). So after you rebase a
branch, @{1} will refer to what previously was ORIG_HEAD.
Having said that, it is an unintended regression in the built-in rebase.
Markus, could you come up with a minimal test case, preferably in the form
of a patch to t/t3415-rebase-autosquash.sh?
Thanks,
Johannes
^ permalink raw reply
* Re: git rebase: retain original head?
From: Markus Wiederkehr @ 2019-01-09 13:48 UTC (permalink / raw)
To: git
In-Reply-To: <87r2dmc1q0.fsf@igel.home>
On Tue, Jan 8, 2019 at 10:29 PM Andreas Schwab <schwab@linux-m68k.org> wrote:
>
> On Jan 08 2019, Markus Wiederkehr <markus.wiederkehr@gmail.com> wrote:
>
> > It is not, depending on what happens in the rebase it could be any
> > entry.
>
> Don't look at the HEAD reflog, use the branch reflog, ie. @{1}.
That's exactly what I was looking for, thanks!
^ permalink raw reply
* Re: [PATCH v3 1/3] t5323: test cases for git-pack-redundant
From: SZEDER Gábor @ 2019-01-09 12:56 UTC (permalink / raw)
To: Jiang Xin; +Cc: Sun Chao, Git List, Junio C Hamano, Jiang Xin
In-Reply-To: <20190102043456.15652-2-worldhello.net@gmail.com>
On Wed, Jan 02, 2019 at 12:34:54PM +0800, Jiang Xin wrote:
> From: Jiang Xin <zhiyou.jx@alibaba-inc.com>
>
> Add test cases for git pack-redundant to validate new algorithm for git
> pack-redundant.
>
> Signed-off-by: Jiang Xin <zhiyou.jx@alibaba-inc.com>
> ---
> t/t5323-pack-redundant.sh | 84 +++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 84 insertions(+)
> create mode 100755 t/t5323-pack-redundant.sh
>
> diff --git a/t/t5323-pack-redundant.sh b/t/t5323-pack-redundant.sh
> new file mode 100755
> index 0000000000..ef6076f065
> --- /dev/null
> +++ b/t/t5323-pack-redundant.sh
> @@ -0,0 +1,84 @@
> +#!/bin/sh
> +#
> +# Copyright (c) 2018 Jiang Xin
> +#
> +
> +test_description='git redundant test'
s/redundant/pack-redundant/ ?
> +
> +. ./test-lib.sh
> +
> +create_commits()
> +{
> + set -e
> + parent=
> + for name in A B C D E F G H I J K L M
> + do
> + test_tick
> + T=$(git write-tree)
> + if test -z "$parent"
> + then
> + sha1=$(echo $name | git commit-tree $T)
There is a considerable effort going on to switch from SHA-1 to a
different hash function, so please don't add any new $sha1 variable;
call it $oid or $commit instead.
> + else
> + sha1=$(echo $name | git commit-tree -p $parent $T)
> + fi
> + eval $name=$sha1
> + parent=$sha1
> + done
> + git update-ref refs/heads/master $M
> +}
> +
> +create_redundant_packs()
> +{
> + set -e
> + cd .git/objects/pack
> + P1=$(printf "$T\n$A\n" | git pack-objects pack 2>/dev/null)
> + P2=$(printf "$T\n$A\n$B\n$C\n$D\n$E\n" | git pack-objects pack 2>/dev/null)
> + P3=$(printf "$C\n$D\n$F\n$G\n$I\n$J\n" | git pack-objects pack 2>/dev/null)
> + P4=$(printf "$D\n$E\n$G\n$H\n$J\n$K\n" | git pack-objects pack 2>/dev/null)
> + P5=$(printf "$F\n$G\n$H\n" | git pack-objects pack 2>/dev/null)
> + P6=$(printf "$F\n$I\n$L\n" | git pack-objects pack 2>/dev/null)
> + P7=$(printf "$H\n$K\n$M\n" | git pack-objects pack 2>/dev/null)
> + P8=$(printf "$L\n$M\n" | git pack-objects pack 2>/dev/null)
> + cd -
> + eval P$P1=P1:$P1
> + eval P$P2=P2:$P2
> + eval P$P3=P3:$P3
> + eval P$P4=P4:$P4
> + eval P$P5=P5:$P5
> + eval P$P6=P6:$P6
> + eval P$P7=P7:$P7
> + eval P$P8=P8:$P8
> +}
> +
> +# Create commits and packs
> +create_commits
> +create_redundant_packs
Please perform all setup tasks in a test_expect_success block, so we
get verbose and trace output about what's going on.
Don't use 'set -e', use an &&-chain instead. To fail the test if a
command in the for loop were to fail you could do something like this:
for ....
do
do-this &&
do-that ||
return 1
done
> +
> +test_expect_success 'clear loose objects' '
> + git prune-packed &&
> + test $(find .git/objects -type f | grep -v pack | wc -l) -eq 0
Use something like
find .git/objects -type f | grep -v pack >out &&
test_must_be_empty out
instead, so we get an informative error message on failure.
> +'
> +
> +cat >expected <<EOF
> +P1:$P1
> +P4:$P4
> +P5:$P5
> +P6:$P6
> +EOF
> +
> +test_expect_success 'git pack-redundant --all' '
> + git pack-redundant --all | \
Don't run a git command (especially the particular command the test
script focuses on) upstream of a pipe, because it hides the command's
exit code. Use an intermediate file instead.
> + sed -e "s#^.*/pack-\(.*\)\.\(idx\|pack\)#\1#g" | \
This sed command doesn't seem to work on macOS (on Travis CI), and
causes the test to fail with:
++git pack-redundant --all
++sed -e 's#^.*/pack-\(.*\)\.\(idx\|pack\)#\1#g'
++sort -u
++read p
++sort
++eval echo '${P.git/objects/pack/pack-0cf5cb6afaa1bae36b8e61ca398dbe29a15bc74e.idx}'
./test-lib.sh: line 697: ${P.git/objects/pack/pack-0cf5cb6afaa1bae36b8e61ca398dbe29a15bc74e.idx}: bad substitution
++test_cmp expected actual
++diff -u expected actual
--- expected 2019-01-09 01:53:45.000000000 +0000
+++ actual 2019-01-09 01:53:45.000000000 +0000
@@ -1,4 +0,0 @@
-P1:24ee080366509364d04a138cd4e168dc4ff33354
-P4:139d8b0cfe7e8970a8f3533835f90278d88de474
-P5:23e0f02d822fa4bfe5ee63337ba5632cd7be208e
-P6:deeb289f1749972f1cd57c3b9f359ece2361f60a
error: last command exited with $?=1
not ok 2 - git pack-redundant --all
I'm not sure what's wrong with it, though.
Minor nit: 'git pack-redundant' prints one filename per line, so the
'g' at the end of the 's###g' is not necessary.
> + sort -u | \
> + while read p; do eval echo "\${P$p}"; done | \
> + sort > actual && \
Style nit: no space between redirection operator and filename
> + test_cmp expected actual
> +'
> +
> +test_expect_success 'remove redundant packs' '
> + git pack-redundant --all | xargs rm &&
> + git fsck &&
> + test $(git pack-redundant --all | wc -l) -eq 0
> +'
> +
> +test_done
> --
> 2.14.5.agit.2
>
^ permalink raw reply
* Re: [PATCH v2 1/1] filter-options: Expand abbreviated numbers
From: SZEDER Gábor @ 2019-01-09 12:23 UTC (permalink / raw)
To: Josh Steadmon; +Cc: git, gitster
In-Reply-To: <d35827de35d2a158cd5325569eaaf355563bf028.1546906008.git.steadmon@google.com>
On Mon, Jan 07, 2019 at 04:17:09PM -0800, Josh Steadmon wrote:
> diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c
> index 5285e7674d..9efb3e9902 100644
> --- a/list-objects-filter-options.c
> +++ b/list-objects-filter-options.c
> @@ -111,6 +112,21 @@ int opt_parse_list_objects_filter(const struct option *opt,
> return parse_list_objects_filter(filter_options, arg);
> }
>
> +void expand_list_objects_filter_spec(
> + const struct list_objects_filter_options *filter,
> + struct strbuf *expanded_spec)
> +{
> + strbuf_init(expanded_spec, strlen(filter->filter_spec));
> + if (filter->choice == LOFC_BLOB_LIMIT)
> + strbuf_addf(expanded_spec, "blob:limit=%lu",
> + filter->blob_limit_value);
> + else if (filter->choice == LOFC_TREE_DEPTH)
> + strbuf_addf(expanded_spec, "tree:%lu",
> + filter->tree_exclude_depth);
> + else
> + strbuf_addstr(expanded_spec, filter->filter_spec);
> +}
> +
All compilers error out with something like this:
list-objects-filter-options.c: In function
‘expand_list_objects_filter_spec’:
list-objects-filter-options.c:124:29: error: ‘LOFC_TREE_DEPTH’ undeclared (first use in this function); did you mean ‘LOFC_TREE_NONE’?
else if (filter->choice == LOFC_TREE_DEPTH)
^~~~~~~~~~~~~~~
LOFC_TREE_NONE
list-objects-filter-options.c:124:29: note: each undeclared identifier is reported only once for each function it appears in
list-objects-filter-options.c:126:14: error: ‘const struct list_objects_filter_options’ has no member named ‘tree_exclude_depth’
filter->tree_exclude_depth);
^~
make: *** [list-objects-filter-options.o] Error 1
^ permalink raw reply
* Re: [PATCH v3 1/5] http: support file handles for HTTP_KEEP_ERROR
From: SZEDER Gábor @ 2019-01-09 12:15 UTC (permalink / raw)
To: Masaya Suzuki; +Cc: git, jrnieder, peff, sunshine
In-Reply-To: <20190108024741.62176-2-masayasuzuki@google.com>
On Mon, Jan 07, 2019 at 06:47:37PM -0800, Masaya Suzuki wrote:
> HTTP_KEEP_ERROR makes it easy to debug HTTP transport errors. In order
> to make HTTP_KEEP_ERROR enabled for all requests, file handles need to
> be supported.
>
> Signed-off-by: Masaya Suzuki <masayasuzuki@google.com>
> ---
> http.c | 9 ++++++---
> 1 file changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/http.c b/http.c
> index 0b6807cef9..06450da96e 100644
> --- a/http.c
> +++ b/http.c
> @@ -1991,16 +1991,19 @@ static int http_request_reauth(const char *url,
> /*
> * If we are using KEEP_ERROR, the previous request may have
> * put cruft into our output stream; we should clear it out before
> - * making our next request. We only know how to do this for
> - * the strbuf case, but that is enough to satisfy current callers.
> + * making our next request.
> */
> if (options && options->keep_error) {
> switch (target) {
> case HTTP_REQUEST_STRBUF:
> strbuf_reset(result);
> break;
> + case HTTP_REQUEST_FILE:
> + fflush(result);
> + ftruncate(fileno(result), 0);
Some GCC versions complain about the above line:
http.c: In function ‘http_request_reauth’:
http.c:1961:3: error: ignoring return value of ‘ftruncate’, declared with attribute warn_unused_result [-Werror=unused-result]
ftruncate(fileno(result), 0);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
CC shell.o
CC remote-testsvn.o
CC vcs-svn/line_buffer.o
cc1: all warnings being treated as errors
make: *** [http.o] Error 1
make: *** Waiting for unfinished jobs....
> + break;
> default:
> - BUG("HTTP_KEEP_ERROR is only supported with strbufs");
> + BUG("Unknown http_request target");
> }
> }
>
> --
> 2.20.1.97.g81188d93c3-goog
>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox