[PATCH] Increase packedGit{Limit,WindowSize} on 64 bit systems.

[PATCH] Increase packedGit{Limit,WindowSize} on 64 bit systems.

[Shawn O. Pearce]

If we have a 64 bit address space we can easily afford to commit
a larger amount of virtual address space to pack file access.
So on these platforms we should increase the default settings of
core.packedGit{Limit,WindowSize} to something that will better
handle very large projects.

Thanks to Andy Whitcroft for pointing out that we can safely
increase these defaults on such systems.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 Documentation/config.txt |   16 ++++++++++------
 git-compat-util.h        |   10 ++++++++--
 2 files changed, 18 insertions(+), 8 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index e4ee52f..b4aae0d 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -125,9 +125,12 @@ core.packedGitWindowSize::
 	more quickly.  Smaller window sizes will negatively affect
 	performance due to increased calls to the operating system's
 	memory manager, but may improve performance when accessing
-	a large number of large pack files.  Default is 32 MiB,
-	which should be reasonable for all users/operating systems.
-	You probably do not need to adjust this value.
+	a large number of large pack files.
++
+Default is 1 MiB if NO_MMAP was set at compile time, otherwise 32
+MiB on 32 bit platforms and 1 GiB on 64 bit platforms.  This should
+be reasonable for all users/operating systems.  You probably do
+not need to adjust this value.
 +
 Common unit suffixes of 'k', 'm', or 'g' are supported.
 
@@ -136,9 +139,10 @@ core.packedGitLimit::
 	from pack files.  If Git needs to access more than this many
 	bytes at once to complete an operation it will unmap existing
 	regions to reclaim virtual address space within the process.
-	Default is 256 MiB, which should be reasonable for all
-	users/operating systems, except on the largest projects.
-	You probably do not need to adjust this value.
++
+Default is 256 MiB on 32 bit platforms and 8 GiB on 64 bit platforms.
+This should be reasonable for all users/operating systems, except on
+the largest projects.  You probably do not need to adjust this value.
 +
 Common unit suffixes of 'k', 'm', or 'g' are supported.
 
diff --git a/git-compat-util.h b/git-compat-util.h
index f243b86..55456da 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -97,11 +97,17 @@ extern int git_munmap(void *start, size_t length);
 #else /* NO_MMAP */
 
 #include <sys/mman.h>
-#define DEFAULT_PACKED_GIT_WINDOW_SIZE (32 * 1024 * 1024)
+#define DEFAULT_PACKED_GIT_WINDOW_SIZE \
+	(sizeof(void*) >= 8 \
+		?  1 * 1024 * 1024 * 1024 \
+		: 32 * 1024 * 1024)
 
 #endif /* NO_MMAP */
 
-#define DEFAULT_PACKED_GIT_LIMIT (256 * 1024 * 1024)
+#define DEFAULT_PACKED_GIT_LIMIT \
+	(sizeof(void*) >= 8 \
+		?   8 * 1024 * 1024 * 1024 \
+		: 256 * 1024 * 1024)
 
 #ifdef NO_SETENV
 #define setenv gitsetenv
-- 
1.5.0.rc0.gce9e

Re: [PATCH] Increase packedGit{Limit,WindowSize} on 64 bit systems.

[Junio C Hamano]

"Shawn O. Pearce" <spearce@spearce.org> writes:

> If we have a 64 bit address space we can easily afford to commit
> a larger amount of virtual address space to pack file access.
> So on these platforms we should increase the default settings of
> core.packedGit{Limit,WindowSize} to something that will better
> handle very large projects.

Hmmmm.  What's the reasoning behind this?

We have more than enough virtual memory anyway, we do not bother
with our own mmap limit -- we will let the operating system
worry about it.

If that is the reasoning, I have a feeling that we might want to
be even more agressive.  If you have a 1.5GB pack, wouldn't you
rather map the whole thing in a single window, instead of
splitting that into two?

Currently we are limited (by pack offset) to 4GB per pack, so
raising the window max to 4GB might make sense.

On the total size of vm space, I am wondering what would happen
if we make this unbounded.  You certainly notice mmap() failure
and fall back to recycle other windows even when your total
usage is under the limit, don't you?  Your later patches in the
series even unmap the mapped but unused window lazily to make
room for xmalloc() and friends, so I suspect it might even make
it simpler not to have this limit especially on larger systems.