[PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects

[PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects

[Jan Krüger]

In 479b56ba ('make "repack -f" imply "pack-objects --no-reuse-object"'),
git repack -f was changed to include recompressing all objects on the
zlib level on the assumption that if the user wants to spend that much
time already, some more time won't hurt (and recompressing is useful if
the user changed the zlib compression level).

However, "some more time" can be quite long with very big repositories,
so some users are going to appreciate being able to choose. Hence, this
adds a new -F option that uses the old behaviour of recalculating deltas
only and keeping the zlib compression intact.

Measurements taken using this patch on a current clone of git.git
indicate a 17% decrease in time being made available to users:

git repack -Adf  38.79s user 0.56s system 133% cpu 29.394 total
git repack -AdF  34.84s user 0.56s system 145% cpu 24.388 total

Signed-off-by: Jan Krüger <jk@jk.gs>
---

The concrete case that prompted me to write this patch was a repository
of 25 GB that some guys were trying to repack. 17% of the time needed to
repack -f that much data is... substantial.

Discussion point: it might make more sense to switch the meanings
around, making -F do the 'bigger' routine and reverting -f to what it
used to be. I don't feel strongly about that, however.

Cc'ing those who were involved in the discussion leading to 479b56ba.

 Documentation/git-repack.txt |    6 +++++-
 git-repack.sh                |    2 ++
 2 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt
index 8c67d17..cce32e2 100644
--- a/Documentation/git-repack.txt
+++ b/Documentation/git-repack.txt
@@ -8,7 +8,7 @@ git-repack - Pack unpacked objects in a repository
 
 SYNOPSIS
 --------
-'git repack' [-a] [-A] [-d] [-f] [-l] [-n] [-q] [--window=N] [--depth=N]
+'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [--window=N] [--depth=N]
 
 DESCRIPTION
 -----------
@@ -65,6 +65,10 @@ other objects in that pack they already have locally.
 	Pass the `--no-reuse-object` option to `git-pack-objects`, see
 	linkgit:git-pack-objects[1].
 
+-F::
+	Pass the `--no-reuse-delta` option to `git-pack-objects`, see
+	linkgit:git-pack-objects[1].
+
 -q::
 	Pass the `-q` option to 'git pack-objects'. See
 	linkgit:git-pack-objects[1].
diff --git a/git-repack.sh b/git-repack.sh
index 1eb3bca..a010406 100755
--- a/git-repack.sh
+++ b/git-repack.sh
@@ -11,6 +11,7 @@ a               pack everything in a single pack
 A               same as -a, and turn unreachable objects loose
 d               remove redundant packs, and run git-prune-packed
 f               pass --no-reuse-object to git-pack-objects
+F               pass --no-reuse-delta to git-pack-objects
 n               do not run git-update-server-info
 q,quiet         be quiet
 l               pass --local to git-pack-objects
@@ -35,6 +36,7 @@ do
 	-d)	remove_redundant=t ;;
 	-q)	GIT_QUIET=t ;;
 	-f)	no_reuse=--no-reuse-object ;;
+	-F)	no_reuse=--no-reuse-delta ;;
 	-l)	local=--local ;;
 	--max-pack-size|--window|--window-memory|--depth)
 		extra="$extra $1=$2"; shift ;;
-- 
1.7.2.3.392.g02377.dirty

Re: [PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects

[Nicolas Pitre]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2066 bytes --]

On Mon, 27 Sep 2010, Jan Krüger wrote:

> In 479b56ba ('make "repack -f" imply "pack-objects --no-reuse-object"'),
> git repack -f was changed to include recompressing all objects on the
> zlib level on the assumption that if the user wants to spend that much
> time already, some more time won't hurt (and recompressing is useful if
> the user changed the zlib compression level).
> 
> However, "some more time" can be quite long with very big repositories,
> so some users are going to appreciate being able to choose. Hence, this
> adds a new -F option that uses the old behaviour of recalculating deltas
> only and keeping the zlib compression intact.
> 
> Measurements taken using this patch on a current clone of git.git
> indicate a 17% decrease in time being made available to users:
> 
> git repack -Adf  38.79s user 0.56s system 133% cpu 29.394 total
> git repack -AdF  34.84s user 0.56s system 145% cpu 24.388 total
> 
> Signed-off-by: Jan Krüger <jk@jk.gs>
> ---
> 
> The concrete case that prompted me to write this patch was a repository
> of 25 GB that some guys were trying to repack. 17% of the time needed to
> repack -f that much data is... substantial.
> 
> Discussion point: it might make more sense to switch the meanings
> around, making -F do the 'bigger' routine and reverting -f to what it
> used to be. I don't feel strongly about that, however.

That's exactly what I was about to propose before I read through your 
email down to this part.

I personally don't find --no-reuse-object particularly useful.  I hardly 
imagine that people are changing the pack compression level that often 
if at all.  So I doubt moving the current --no-reuse-object behavior to 
-F and reverting -f to  --no-reuse-delta would cause any serious 
inconvenience.  It certainly won't _break_ anything.  So you have my ACK 
to do that change.

In addition to that change, perhaps a note could be added to the 
documentation for pack.compression indicating that for the new setting 
to take effect for existing packs, they must be repacked with -F.


Nicolas

[Alt. PATCH 1/2] repack: add -F flag to let user choose between --no-reuse-delta/object

[Jan Krüger]

In 479b56ba ('make "repack -f" imply "pack-objects --no-reuse-object"'),
git repack -f was changed to include recompressing all objects on the
zlib level on the assumption that if the user wants to spend that much
time already, some more time won't hurt (and recompressing is useful if
the user changed the zlib compression level).

However, "some more time" can be quite long with very big repositories,
so some users are going to appreciate being able to choose. If we are
going to give them the choice, --no-reuse-object will probably be
interesting a lot less frequently than --no-reuse-delta. Hence, this
reverts -f to the old behaviour (--no-reuse-delta) and adds a new -F
option that replaces the current -f.

Measurements taken using this patch on a current clone of git.git
indicate a 17% decrease in time being made available to users:

git repack -Adf  34.84s user 0.56s system 145% cpu 24.388 total
git repack -AdF  38.79s user 0.56s system 133% cpu 29.394 total

Signed-off-by: Jan Krüger <jk@jk.gs>
---

--- Nicolas Pitre <nico@fluxnic.net> wrote:

> I personally don't find --no-reuse-object particularly useful.  I
> hardly imagine that people are changing the pack compression level
> that often if at all.  So I doubt moving the current
> --no-reuse-object behavior to -F and reverting -f to
> --no-reuse-delta would cause any serious inconvenience.  It certainly
> won't _break_ anything.  So you have my ACK to do that change.

Here we go.

 Documentation/git-repack.txt |    6 +++++-
 git-repack.sh                |    6 ++++--
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt
index 8c67d17..9566727 100644
--- a/Documentation/git-repack.txt
+++ b/Documentation/git-repack.txt
@@ -8,7 +8,7 @@ git-repack - Pack unpacked objects in a repository
 
 SYNOPSIS
 --------
-'git repack' [-a] [-A] [-d] [-f] [-l] [-n] [-q] [--window=N] [--depth=N]
+'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [--window=N] [--depth=N]
 
 DESCRIPTION
 -----------
@@ -62,6 +62,10 @@ other objects in that pack they already have locally.
 	linkgit:git-pack-objects[1].
 
 -f::
+	Pass the `--no-reuse-delta` option to `git-pack-objects`, see
+	linkgit:git-pack-objects[1].
+
+-F::
 	Pass the `--no-reuse-object` option to `git-pack-objects`, see
 	linkgit:git-pack-objects[1].
 
diff --git a/git-repack.sh b/git-repack.sh
index 1eb3bca..769baaf 100755
--- a/git-repack.sh
+++ b/git-repack.sh
@@ -10,7 +10,8 @@ git repack [options]
 a               pack everything in a single pack
 A               same as -a, and turn unreachable objects loose
 d               remove redundant packs, and run git-prune-packed
-f               pass --no-reuse-object to git-pack-objects
+f               pass --no-reuse-delta to git-pack-objects
+F               pass --no-reuse-object to git-pack-objects
 n               do not run git-update-server-info
 q,quiet         be quiet
 l               pass --local to git-pack-objects
@@ -34,7 +35,8 @@ do
 		unpack_unreachable=--unpack-unreachable ;;
 	-d)	remove_redundant=t ;;
 	-q)	GIT_QUIET=t ;;
-	-f)	no_reuse=--no-reuse-object ;;
+	-f)	no_reuse=--no-reuse-delta ;;
+	-F)	no_reuse=--no-reuse-object ;;
 	-l)	local=--local ;;
 	--max-pack-size|--window|--window-memory|--depth)
 		extra="$extra $1=$2"; shift ;;
-- 
1.7.2.3.392.g02377.dirty

[Alt. PATCH 2/2] Documentation: pack.compression: explain how to recompress

[Jan Krüger]

Add a small remark about how to recompress all existing objects after
changing the compression level for pack files.

Signed-off-by: Jan Krüger <jk@jk.gs>
---

--- Nicolas Pitre <nico@fluxnic.net> wrote:

> In addition to that change, perhaps a note could be added to the 
> documentation for pack.compression indicating that for the new
> setting to take effect for existing packs, they must be repacked with
> -F.

Good idea. Here it is.

 Documentation/config.txt |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index d294dd6..506477b 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -1465,6 +1465,10 @@ pack.compression::
 	not set,  defaults to -1, the zlib default, which is "a default
 	compromise between speed and compression (currently equivalent
 	to level 6)."
++
+Note that changing the compression level will not automatically recompress
+all existing objects. You can force recompression by passing the -F option
+to linkgit:git-repack[1].
 
 pack.deltaCacheSize::
 	The maximum memory in bytes used for caching deltas in
-- 
1.7.2.3.392.g02377.dirty

Re: [Alt. PATCH 2/2] Documentation: pack.compression: explain how to recompress

[Nicolas Pitre]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1525 bytes --]

On Mon, 27 Sep 2010, Jan Krüger wrote:

> Add a small remark about how to recompress all existing objects after
> changing the compression level for pack files.
> 
> Signed-off-by: Jan Krüger <jk@jk.gs>

Acked-by: Nicolas Pitre <nico@fluxnic.net>


> ---
> 
> --- Nicolas Pitre <nico@fluxnic.net> wrote:
> 
> > In addition to that change, perhaps a note could be added to the 
> > documentation for pack.compression indicating that for the new
> > setting to take effect for existing packs, they must be repacked with
> > -F.
> 
> Good idea. Here it is.
> 
>  Documentation/config.txt |    4 ++++
>  1 files changed, 4 insertions(+), 0 deletions(-)
> 
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index d294dd6..506477b 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -1465,6 +1465,10 @@ pack.compression::
>  	not set,  defaults to -1, the zlib default, which is "a default
>  	compromise between speed and compression (currently equivalent
>  	to level 6)."
> ++
> +Note that changing the compression level will not automatically recompress
> +all existing objects. You can force recompression by passing the -F option
> +to linkgit:git-repack[1].
>  
>  pack.deltaCacheSize::
>  	The maximum memory in bytes used for caching deltas in
> -- 
> 1.7.2.3.392.g02377.dirty
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [Alt. PATCH 1/2] repack: add -F flag to let user choose between --no-reuse-delta/object

[Nicolas Pitre]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3815 bytes --]

On Mon, 27 Sep 2010, Jan Krüger wrote:

> In 479b56ba ('make "repack -f" imply "pack-objects --no-reuse-object"'),
> git repack -f was changed to include recompressing all objects on the
> zlib level on the assumption that if the user wants to spend that much
> time already, some more time won't hurt (and recompressing is useful if
> the user changed the zlib compression level).
> 
> However, "some more time" can be quite long with very big repositories,
> so some users are going to appreciate being able to choose. If we are
> going to give them the choice, --no-reuse-object will probably be
> interesting a lot less frequently than --no-reuse-delta. Hence, this
> reverts -f to the old behaviour (--no-reuse-delta) and adds a new -F
> option that replaces the current -f.
> 
> Measurements taken using this patch on a current clone of git.git
> indicate a 17% decrease in time being made available to users:
> 
> git repack -Adf  34.84s user 0.56s system 145% cpu 24.388 total
> git repack -AdF  38.79s user 0.56s system 133% cpu 29.394 total
> 
> Signed-off-by: Jan Krüger <jk@jk.gs>

Acked-by: Nicolas Pitre <nico@fluxnic.net>


> ---
> 
> --- Nicolas Pitre <nico@fluxnic.net> wrote:
> 
> > I personally don't find --no-reuse-object particularly useful.  I
> > hardly imagine that people are changing the pack compression level
> > that often if at all.  So I doubt moving the current
> > --no-reuse-object behavior to -F and reverting -f to
> > --no-reuse-delta would cause any serious inconvenience.  It certainly
> > won't _break_ anything.  So you have my ACK to do that change.
> 
> Here we go.
> 
>  Documentation/git-repack.txt |    6 +++++-
>  git-repack.sh                |    6 ++++--
>  2 files changed, 9 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/git-repack.txt b/Documentation/git-repack.txt
> index 8c67d17..9566727 100644
> --- a/Documentation/git-repack.txt
> +++ b/Documentation/git-repack.txt
> @@ -8,7 +8,7 @@ git-repack - Pack unpacked objects in a repository
>  
>  SYNOPSIS
>  --------
> -'git repack' [-a] [-A] [-d] [-f] [-l] [-n] [-q] [--window=N] [--depth=N]
> +'git repack' [-a] [-A] [-d] [-f] [-F] [-l] [-n] [-q] [--window=N] [--depth=N]
>  
>  DESCRIPTION
>  -----------
> @@ -62,6 +62,10 @@ other objects in that pack they already have locally.
>  	linkgit:git-pack-objects[1].
>  
>  -f::
> +	Pass the `--no-reuse-delta` option to `git-pack-objects`, see
> +	linkgit:git-pack-objects[1].
> +
> +-F::
>  	Pass the `--no-reuse-object` option to `git-pack-objects`, see
>  	linkgit:git-pack-objects[1].
>  
> diff --git a/git-repack.sh b/git-repack.sh
> index 1eb3bca..769baaf 100755
> --- a/git-repack.sh
> +++ b/git-repack.sh
> @@ -10,7 +10,8 @@ git repack [options]
>  a               pack everything in a single pack
>  A               same as -a, and turn unreachable objects loose
>  d               remove redundant packs, and run git-prune-packed
> -f               pass --no-reuse-object to git-pack-objects
> +f               pass --no-reuse-delta to git-pack-objects
> +F               pass --no-reuse-object to git-pack-objects
>  n               do not run git-update-server-info
>  q,quiet         be quiet
>  l               pass --local to git-pack-objects
> @@ -34,7 +35,8 @@ do
>  		unpack_unreachable=--unpack-unreachable ;;
>  	-d)	remove_redundant=t ;;
>  	-q)	GIT_QUIET=t ;;
> -	-f)	no_reuse=--no-reuse-object ;;
> +	-f)	no_reuse=--no-reuse-delta ;;
> +	-F)	no_reuse=--no-reuse-object ;;
>  	-l)	local=--local ;;
>  	--max-pack-size|--window|--window-memory|--depth)
>  		extra="$extra $1=$2"; shift ;;
> -- 
> 1.7.2.3.392.g02377.dirty
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects

[Junio C Hamano]

Jan Krüger <jk@jk.gs> writes:

> Discussion point: it might make more sense to switch the meanings
> around, making -F do the 'bigger' routine and reverting -f to what it
> used to be. I don't feel strongly about that, however.

That sounds sensible.  reuse_object is used only to z-recompress and is
not involved in precomputed delta selection, and under normal
circumstances it should not have much effect on the outcome.

Please make it so.

Re: [PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects

[Jan Krüger]

--- Junio C Hamano <gitster@pobox.com> wrote:

> Jan Krüger <jk@jk.gs> writes:
> 
> > Discussion point: it might make more sense to switch the meanings
> > around, making -F do the 'bigger' routine and reverting -f to what
> > it used to be. I don't feel strongly about that, however.
> 
> That sounds sensible.  reuse_object is used only to z-recompress and
> is not involved in precomputed delta selection, and under normal
> circumstances it should not have much effect on the outcome.
> 
> Please make it so.

Already done. See the alternative patch (in a 2-series) at
20100927141936.590d71b3@jk.gs
<http://mid.gmane.org/20100927141936.590d71b3@jk.gs>. I Cc'd you and
Nicolas acked it. Just saying. ;-)

-Jan

Re: [PATCH] repack: add -F option that passes --no-reuse-delta to pack-objects

[Junio C Hamano]

Jan Krüger <jk@jk.gs> writes:

>> Please make it so.
>
> Already done.

Yeah, I saw and queued them ;-)

Thanks.