* Re: Creating objects manually and repack
From: Linus Torvalds @ 2006-08-04 16:53 UTC (permalink / raw)
To: Jon Smirl; +Cc: git
In-Reply-To: <9e4733910608040841v7f4f27efra63e5ead2656e07@mail.gmail.com>
On Fri, 4 Aug 2006, Jon Smirl wrote:
>
> How about forking off a pack-objects and handing it one file name at a
> time over a pipe. When I hand it the next file name I delete the first
> file. Does pack-objects make multiple passes over the files? This
> model would let me hand it all 1M files.
pack-objects does actually make several (well, two) passes over the
objects right now, because it first does all the sorting based on object
size/type, and then does the actual deltifying pass.
But doing things one file-name at a time would certainly be fine. You can
even do it with git-pack-objects running in parallel, ie you can do a
for_each_filename() {
cvs-generate-objects filename | git-pack-objects filename
rm -rf .git/objects/??/
}
and then "cvs-generate-objects" should just make sure that it writes the
git object _before_ it actually outputs the object name on stdout.
And if you do it this way, you won't even have to pass any filenames,
since git-pack-objects will only get objects for the same file, and will
do the right thing just sorting them by size.
So in the above kind of setting, the _only_ thing that
cvs-generate-objects needs to do is:
for_each_rev(file) {
unsigned char sha1[20];
unsigned long len;
void *buf;
/* unpack the revision into memory */
buf = cvs_unpack_revision(&len);
/* Write it out as a git blob file */
write_sha1_file(buf, len, "blob", sha1);
/* Free the memory image */
free(buf);
/* Tell git-pack-objects the name of the git blob */
printf("%s\n", sha1_to_hex(sha1));
}
and you're basically all done. The above would turn each *,v file into a
*-<sha>.pack/*-<sha>.idx file pair, so you'd have exactly as many
pack-files as you have *,v files.
Linus
^ permalink raw reply
* Re: Creating objects manually and repack
From: Rogan Dawes @ 2006-08-04 16:39 UTC (permalink / raw)
To: Jon Smirl; +Cc: git
In-Reply-To: <9e4733910608040841v7f4f27efra63e5ead2656e07@mail.gmail.com>
Jon Smirl wrote:
> On 8/4/06, Linus Torvalds <torvalds@osdl.org> wrote:
>> I'd suggest against it, but you can (and should) just repack often enough
>> that you shouldn't ever have gigabytes of objects "in flight". I'd have
>> expected that with a repack every few ten thousand files, and most files
>> being on the order of a few kB, you'd have been more than ok, but
>> especially if you have large files, you may want to make things "every
>> <n>
>> bytes" rather than "every <n> files".
>
> How about forking off a pack-objects and handing it one file name at a
> time over a pipe. When I hand it the next file name I delete the first
> file. Does pack-objects make multiple passes over the files? This
> model would let me hand it all 1M files.
>
I'd imagine that this would not necessarily save you a lot, if you have
to write it to disk, and then read it back again. Your only chance here
is if you stay in the buffer, and avoid actually writing to disk at all.
Of course, using a ramdisk/tmpfs for your object directories might be
enough to save you. Just use a symlink to tmpfs for the objects
directory, and leave the pack files on persistent storage.
That doesn't answer your question about how many passes pack-objects
does. Nicholas Pitre should be able to answer that.
Rogan
^ permalink raw reply
* Re: Creating objects manually and repack
From: Linus Torvalds @ 2006-08-04 16:32 UTC (permalink / raw)
To: Jon Smirl; +Cc: gitzilla, git
In-Reply-To: <9e4733910608040911p443a1360k6d9d1aab00039100@mail.gmail.com>
On Fri, 4 Aug 2006, Jon Smirl wrote:
>
> That is under consideration but the undeltafied pack is about 12GB and
> it takes forever (about a day) to deltafy it. I'm not convinced yet
> that an undeltafied pack is any faster than just having the objects in
> the directories.
Yeah, I think it's worth it deltifying things early, as you seem to get
all the object info in the right order anyway (ie you do the revisions for
one file in one go).
Linus
^ permalink raw reply
* Re: [KORG] kernel.org/git/ showing nothing
From: H. Peter Anvin @ 2006-08-04 16:25 UTC (permalink / raw)
To: Jeff Garzik; +Cc: ftpadmin, Git Mailing List
In-Reply-To: <44D3208E.8090403@garzik.org>
Jeff Garzik wrote:
> When I visit http://www.kernel.org/git/ no projects at all are listed.
>
> At least one other person independently noted this, as well.
>
Bloody Hades.
I swear if I ever meet the guy who decided that FC5 didn't need to be
binary compatible with FC4 I will personally kill him with my bare hands...
Fixed now.
-hpa
^ permalink raw reply
* [PATCH] merge-recursive: fix rename handling
From: Johannes Schindelin @ 2006-08-04 16:21 UTC (permalink / raw)
To: git, junkio
To handle renames properly, we iterate through all file names of both
heads, the current one, and the one to be merged.
Only that there was a bug, where it was checked if the file name was present
in both heads, but the result of the check was never used. Instead, the
merge proceeded as if both heads contained that file.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
merge-recursive.c | 6 ++++--
1 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/merge-recursive.c b/merge-recursive.c
index 10bce70..1e176ca 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -808,8 +808,10 @@ static int process_renames(struct path_l
} else {
compare = strcmp(a_renames->items[i].path,
b_renames->items[j].path);
- ren1 = a_renames->items[i++].util;
- ren2 = b_renames->items[j++].util;
+ if (compare <= 0)
+ ren1 = a_renames->items[i++].util;
+ if (compare >= 0)
+ ren2 = b_renames->items[j++].util;
}
/* TODO: refactor, so that 1/2 are not needed */
--
1.4.2.rc2.ga8a2
^ permalink raw reply related
* Re: Creating objects manually and repack
From: Jon Smirl @ 2006-08-04 16:11 UTC (permalink / raw)
To: gitzilla; +Cc: git, Linus Torvalds
In-Reply-To: <44D36F64.5040404@gmail.com>
On 8/4/06, A Large Angry SCM <gitzilla@gmail.com> wrote:
> Jon Smirl wrote:
> > On 8/4/06, Linus Torvalds <torvalds@osdl.org> wrote:
> >> I'd suggest against it, but you can (and should) just repack often enough
> >> that you shouldn't ever have gigabytes of objects "in flight". I'd have
> >> expected that with a repack every few ten thousand files, and most files
> >> being on the order of a few kB, you'd have been more than ok, but
> >> especially if you have large files, you may want to make things "every
> >> <n>
> >> bytes" rather than "every <n> files".
> >
> > How about forking off a pack-objects and handing it one file name at a
> > time over a pipe. When I hand it the next file name I delete the first
> > file. Does pack-objects make multiple passes over the files? This
> > model would let me hand it all 1M files.
> >
>
> Why don't you just write the pack file directly? Pack files without
> deltas have a very simple structure, and git-index-pack will create a
> pack index file for the pack file you give it.
That is under consideration but the undeltafied pack is about 12GB and
it takes forever (about a day) to deltafy it. I'm not convinced yet
that an undeltafied pack is any faster than just having the objects in
the directories.
The same data in a deltafied pack is 700MB. That is a tremendous
difference in the amount of IO needed. The strategy has to be to avoid
IO, nothing I am doing is ever CPU bound.
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: Creating objects manually and repack
From: A Large Angry SCM @ 2006-08-04 16:01 UTC (permalink / raw)
To: git; +Cc: Jon Smirl, Linus Torvalds
In-Reply-To: <9e4733910608040841v7f4f27efra63e5ead2656e07@mail.gmail.com>
Jon Smirl wrote:
> On 8/4/06, Linus Torvalds <torvalds@osdl.org> wrote:
>> I'd suggest against it, but you can (and should) just repack often enough
>> that you shouldn't ever have gigabytes of objects "in flight". I'd have
>> expected that with a repack every few ten thousand files, and most files
>> being on the order of a few kB, you'd have been more than ok, but
>> especially if you have large files, you may want to make things "every
>> <n>
>> bytes" rather than "every <n> files".
>
> How about forking off a pack-objects and handing it one file name at a
> time over a pipe. When I hand it the next file name I delete the first
> file. Does pack-objects make multiple passes over the files? This
> model would let me hand it all 1M files.
>
Why don't you just write the pack file directly? Pack files without
deltas have a very simple structure, and git-index-pack will create a
pack index file for the pack file you give it.
^ permalink raw reply
* [PATCH 3/4] autoconf: Typo cleanup, reordering etc.
From: Jakub Narebski @ 2006-08-04 15:55 UTC (permalink / raw)
To: git; +Cc: Jakub Narebski
In-Reply-To: <7v7j1on71n.fsf@assigned-by-dhcp.cox.net>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
Makefile | 2 +-
configure.ac | 43 +++++++++++++++++++++++--------------------
2 files changed, 24 insertions(+), 21 deletions(-)
diff --git a/Makefile b/Makefile
index d662bd6..c6b62d9 100644
--- a/Makefile
+++ b/Makefile
@@ -22,7 +22,7 @@ #
# Define NO_C99_FORMAT if your formatted IO functions (printf/scanf et.al.)
# do not support the 'size specifiers' introduced by C99, namely ll, hh,
# j, z, t. (representing long long int, char, intmax_t, size_t, ptrdiff_t).
-# some c compilers supported these specifiers prior to C99 as an extension.
+# some C compilers supported these specifiers prior to C99 as an extension.
#
# Define NO_STRCASESTR if you don't have strcasestr.
#
diff --git a/configure.ac b/configure.ac
index 1796cf4..a88219a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -51,7 +51,7 @@ fi; \
## Site configuration
## --with-PACKAGE[=ARG] and --without-PACKAGE
#
-# Define NO_SVN_TESTS if you want to skip time-consuming SVN interopability
+# Define NO_SVN_TESTS if you want to skip time-consuming SVN interoperability
# tests. These tests take up a significant amount of the total test time
# but are not needed unless you plan to talk to SVN repos.
#
@@ -81,7 +81,24 @@ # not built, and you cannot push using h
#
# Define NO_MMAP if you want to avoid mmap.
#
-# Define NO_PYTHON if you want to loose all benefits of the recursive merge.
+# Define SHELL_PATH to provide path to shell.
+GIT_ARG_SET_PATH(shell)
+#
+# Define PERL_PATH to provide path to Perl.
+GIT_ARG_SET_PATH(perl)
+#
+# Define NO_PYTHON if you want to lose all benefits of the recursive merge.
+# Define PYTHON_PATH to provide path to Python.
+AC_ARG_WITH(python,[AS_HELP_STRING([--with-python=PATH], [provide PATH to python])
+AS_HELP_STRING([--no-python], [don't use python scripts])],
+ [if test "$withval" = "no"; then \
+ NO_PYTHON=YesPlease; \
+ elif test "$withval" != "yes"; then \
+ PYTHON_PATH=$withval; \
+ fi; \
+ ])
+AC_SUBST(NO_PYTHON)
+AC_SUBST(PYTHON_PATH)
#
## --enable-FEATURE[=ARG] and --disable-FEATURE
# Define COLLISION_CHECK below if you believe that SHA1's
@@ -101,27 +118,13 @@ # change being considered an inode chang
## Checks for programs.
AC_MSG_NOTICE([CHECKS for programs])
#
-GIT_ARG_SET_PATH(shell)
-GIT_ARG_SET_PATH(perl)
-AC_ARG_WITH(python,[AS_HELP_STRING([--with-python=PATH], [provide PATH to python])
-AS_HELP_STRING([--no-python], [don't use python scripts])],
- [if test "$withval" = "no"; then \
- NO_PYTHON=YesPlease; \
- elif test "$withval" != "yes"; then \
- PYTHON_PATH=$withval; \
- fi; \
- ])
-AC_SUBST(NO_PYTHON)
-AC_SUBST(PYTHON_PATH)
-
-
-#
-# Define NO_PYTHON if you want to lose all benefits of the recursive merge.
-# Define PYTHON_PATH to provide path to Python.
AC_PROG_CC
#AC_PROG_INSTALL # needs install-sh or install.sh in sources
AC_CHECK_TOOL(AR, ar, :)
AC_CHECK_PROGS(TAR, [gtar tar])
+#
+# Define NO_PYTHON if you want to lose all benefits of the recursive merge.
+# Define PYTHON_PATH to provide path to Python.
if test -z "$NO_PYTHON"; then
AC_PATH_PROGS(PYTHON_PATH, [python2.4 python2.3 python2 python])
if test -n "$PYTHON_PATH"; then
@@ -194,7 +197,7 @@ # Define NO_C99_FORMAT if your formatted
# do not support the 'size specifiers' introduced by C99, namely ll, hh,
# j, z, t. (representing long long int, char, intmax_t, size_t, ptrdiff_t).
# some C compilers supported these specifiers prior to C99 as an extension.
-AC_CACHE_CHECK(whether IO functions support %ll %hh %j %z %t size specifiers,
+AC_CACHE_CHECK(whether formatted IO functions support C99 size specifiers,
ac_cv_c_c99_format,
[# Actually git uses only %z (%zu) in alloc.c, and %t (%td) in mktag.c
AC_RUN_IFELSE(
--
1.4.1.1
^ permalink raw reply related
* [PATCH 1/4] autoconf: Check for working mmap
From: Jakub Narebski @ 2006-08-04 15:55 UTC (permalink / raw)
To: git; +Cc: Jakub Narebski
In-Reply-To: <7v7j1on71n.fsf@assigned-by-dhcp.cox.net>
Use AC_FUNC_MMAP check to check if the `mmap' function exists and
works correctly. (It only checks private fixed mapping of
already-mapped memory.)
Attention: uses implementation detail of AC_FUNC_MMAP!
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
configure.ac | 4 ++++
1 files changed, 4 insertions(+), 0 deletions(-)
diff --git a/configure.ac b/configure.ac
index 0a54b44..178220f 100644
--- a/configure.ac
+++ b/configure.ac
@@ -208,6 +208,10 @@ AC_CHECK_FUNC(setenv,[],
[GIT_CONF_APPEND_LINE(NO_SETENV=YesPlease)])
#
# Define NO_MMAP if you want to avoid mmap.
+AC_FUNC_MMAP
+if test $ac_cv_func_mmap_fixed_mapped != yes; then
+ GIT_CONF_APPEND_LINE(NO_MMAP=YesPlease)
+fi
#
# Define NO_IPV6 if you lack IPv6 support and getaddrinfo().
#
--
1.4.1.1
^ permalink raw reply related
* [PATCH 2/4] autoconf: Check for ll hh j z t size specifiers introduced by C99
From: Jakub Narebski @ 2006-08-04 15:55 UTC (permalink / raw)
To: git; +Cc: Jakub Narebski
In-Reply-To: <7v7j1on71n.fsf@assigned-by-dhcp.cox.net>
Add custom test for checking whether formatted IO functions
(printf/scanf et.al.) support 'size specifiers' introduced by C99,
namely ll, hh, j, z, t. (representing long long int, char, intmax_t,
size_t, ptrdiff_t).
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
configure.ac | 21 +++++++++++++++++++++
1 files changed, 21 insertions(+), 0 deletions(-)
diff --git a/configure.ac b/configure.ac
index 178220f..1796cf4 100644
--- a/configure.ac
+++ b/configure.ac
@@ -189,6 +189,27 @@ # sockaddr_storage.
AC_CHECK_TYPE(struct sockaddr_storage,[],
[GIT_CONF_APPEND_LINE(NO_SOCKADDR_STORAGE=YesPlease)],
[#include <netinet/in.h>])
+#
+# Define NO_C99_FORMAT if your formatted IO functions (printf/scanf et.al.)
+# do not support the 'size specifiers' introduced by C99, namely ll, hh,
+# j, z, t. (representing long long int, char, intmax_t, size_t, ptrdiff_t).
+# some C compilers supported these specifiers prior to C99 as an extension.
+AC_CACHE_CHECK(whether IO functions support %ll %hh %j %z %t size specifiers,
+ ac_cv_c_c99_format,
+[# Actually git uses only %z (%zu) in alloc.c, and %t (%td) in mktag.c
+AC_RUN_IFELSE(
+ [AC_LANG_PROGRAM([AC_INCLUDES_DEFAULT],
+ [[char buf[64];
+ if (sprintf(buf, "%lld%hhd%jd%zd%td", (long long int)1, (char)2, (intmax_t)3, (size_t)4, (ptrdiff_t)5) != 5)
+ exit(1);
+ else if (strcmp(buf, "12345"))
+ exit(2);]])],
+ [ac_cv_c_c99_format=yes],
+ [ac_cv_c_c99_format=no])
+])
+if test $ac_cv_c_c99_format = no; then
+ GIT_CONF_APPEND_LINE(NO_C99_FORMAT=YesPlease)
+fi
## Checks for library functions.
--
1.4.1.1
^ permalink raw reply related
* [PATCH 4/4] Copy description of new build configuration variables to configure.ac
From: Jakub Narebski @ 2006-08-04 15:55 UTC (permalink / raw)
To: git; +Cc: Jakub Narebski
In-Reply-To: <7v7j1on71n.fsf@assigned-by-dhcp.cox.net>
Copy description of new build configuration variables from the
commentary in the top Makefile, namely NO_FINK and NO_DARWIN_PORTS
configuration variables, putting them in site configuration section.
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
configure.ac | 12 ++++++++++++
1 files changed, 12 insertions(+), 0 deletions(-)
diff --git a/configure.ac b/configure.ac
index a88219a..76bfa9d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -79,6 +79,18 @@ #
# Define NO_EXPAT if you do not have expat installed. git-http-push is
# not built, and you cannot push using http:// and https:// transports.
#
+# Define NO_FINK if you are building on Darwin/Mac OS X, have Fink
+# installed in /sw, but don't want GIT to link against any libraries
+# installed there. If defined you may specify your own (or Fink's)
+# include directories and library directories by defining CFLAGS
+# and LDFLAGS appropriately.
+#
+# Define NO_DARWIN_PORTS if you are building on Darwin/Mac OS X,
+# have DarwinPorts installed in /opt/local, but don't want GIT to
+# link against any libraries installed there. If defined you may
+# specify your own (or DarwinPort's) include directories and
+# library directories by defining CFLAGS and LDFLAGS appropriately.
+#
# Define NO_MMAP if you want to avoid mmap.
#
# Define SHELL_PATH to provide path to shell.
--
1.4.1.1
^ permalink raw reply related
* [PATCH] http-push: avoid fork() by calling merge_bases() directly
From: Johannes Schindelin @ 2006-08-04 15:50 UTC (permalink / raw)
To: git, junkio
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
---
http-push.c | 49 +++++++------------------------------------------
1 files changed, 7 insertions(+), 42 deletions(-)
diff --git a/http-push.c b/http-push.c
index 4021e7d..dc82657 100644
--- a/http-push.c
+++ b/http-push.c
@@ -2182,49 +2182,14 @@ static void fetch_symref(const char *pat
static int verify_merge_base(unsigned char *head_sha1, unsigned char *branch_sha1)
{
- int pipe_fd[2];
- pid_t merge_base_pid;
- char line[PATH_MAX + 20];
- unsigned char merge_sha1[20];
- int verified = 0;
-
- if (pipe(pipe_fd) < 0)
- die("Verify merge base: pipe failed");
-
- merge_base_pid = fork();
- if (!merge_base_pid) {
- static const char *args[] = {
- "merge-base",
- "-a",
- NULL,
- NULL,
- NULL
- };
- args[2] = strdup(sha1_to_hex(head_sha1));
- args[3] = sha1_to_hex(branch_sha1);
-
- dup2(pipe_fd[1], 1);
- close(pipe_fd[0]);
- close(pipe_fd[1]);
- execv_git_cmd(args);
- die("merge-base setup failed");
- }
- if (merge_base_pid < 0)
- die("merge-base fork failed");
-
- dup2(pipe_fd[0], 0);
- close(pipe_fd[0]);
- close(pipe_fd[1]);
- while (fgets(line, sizeof(line), stdin) != NULL) {
- if (get_sha1_hex(line, merge_sha1))
- die("expected sha1, got garbage:\n %s", line);
- if (!memcmp(branch_sha1, merge_sha1, 20)) {
- verified = 1;
- break;
- }
- }
+ struct commit *head = lookup_commit(head_sha1);
+ struct commit *branch = lookup_commit(branch_sha1);
+ struct commit_list *merge_bases = get_merge_bases(head, branch, 1);
- return verified;
+ if (merge_bases && !merge_bases->next && merge_bases->item == branch)
+ return 1;
+
+ return 0;
}
static int delete_remote_branch(char *pattern, int force)
--
1.4.2.rc2.g0bca-dirty
^ permalink raw reply related
* [PATCH] Fix crash when GIT_DIR is invalid
From: Johannes Schindelin @ 2006-08-04 15:46 UTC (permalink / raw)
To: git, junkio
We used to test if a pointer was NULL, and if it was, try to access it.
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
---
setup.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/setup.c b/setup.c
index 358e139..2afdba4 100644
--- a/setup.c
+++ b/setup.c
@@ -184,7 +184,7 @@ const char *setup_git_directory_gently(i
}
return NULL;
bad_dir_environ:
- if (!nongit_ok) {
+ if (nongit_ok) {
*nongit_ok = 1;
return NULL;
}
--
1.4.2.rc2.g0bca-dirty
^ permalink raw reply related
* Re: Creating objects manually and repack
From: Jon Smirl @ 2006-08-04 15:41 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <Pine.LNX.4.64.0608040818270.5167@g5.osdl.org>
On 8/4/06, Linus Torvalds <torvalds@osdl.org> wrote:
> I'd suggest against it, but you can (and should) just repack often enough
> that you shouldn't ever have gigabytes of objects "in flight". I'd have
> expected that with a repack every few ten thousand files, and most files
> being on the order of a few kB, you'd have been more than ok, but
> especially if you have large files, you may want to make things "every <n>
> bytes" rather than "every <n> files".
How about forking off a pack-objects and handing it one file name at a
time over a pipe. When I hand it the next file name I delete the first
file. Does pack-objects make multiple passes over the files? This
model would let me hand it all 1M files.
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: Creating objects manually and repack
From: Linus Torvalds @ 2006-08-04 15:22 UTC (permalink / raw)
To: Jon Smirl; +Cc: git
In-Reply-To: <9e4733910608040750g3f72c07ct43f54347e47f25b4@mail.gmail.com>
On Fri, 4 Aug 2006, Jon Smirl wrote:
>
> Could repack-objects be modified to take the objects on stdin as I
> generate them instead of me putting them into the file system and then
> deleting them? That model would avoid many gigabytes of IO.
I'd suggest against it, but you can (and should) just repack often enough
that you shouldn't ever have gigabytes of objects "in flight". I'd have
expected that with a repack every few ten thousand files, and most files
being on the order of a few kB, you'd have been more than ok, but
especially if you have large files, you may want to make things "every <n>
bytes" rather than "every <n> files".
You _could_ also decide to create packs very aggressively indeed, and if
you do them quickly enough, the raw objects never even get written back to
disk before you delete them. That will leave you with a lot of packs, but
you could then "repack the packs" every once in a while.
That said, it's obviously not _impossible_ to do what you suggest, it's
just major surgery to pack-objects (which I'm not going to have time to
do, since I'll be going on a vacation this weekend).
Linus
^ permalink raw reply
* Re: Creating objects manually and repack
From: Jon Smirl @ 2006-08-04 14:50 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <9e4733910608040740x23a8b0cs3bc276ef9e6fb8f7@mail.gmail.com>
The whole problem with CVS import is avoiding getting IO bound. Since
Mozilla CVS expands into 20GB when the revisions are separated out
doing all that IO takes a lot of time. When these imports take four
days it is all IO time, not CPU.
Could repack-objects be modified to take the objects on stdin as I
generate them instead of me putting them into the file system and then
deleting them? That model would avoid many gigabytes of IO.
It might work to just stream the output from zlib into repack-objects
and let it recompute the object name. Or could I just stream in the
uncompressed objects? I can still compute the object sha name in my
code so that I can find it later.
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: Creating objects manually and repack
From: Jon Smirl @ 2006-08-04 14:40 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <Pine.LNX.4.64.0608032150510.4168@g5.osdl.org>
One thing is obvious, I need to tune the repacks to happen before
things spill out of the cache. git repack-objects has been chugging
away for 2hrs now at 2% CPU and 3000 io/sec. It is in one of those
modes where it went back to get the early stuff and in the process of
getting that it knocked the later stuff out of the cache basically
rendering the cache useless.
I'm making good progress with this. I have hit two bugs in cvs2svn
that I will need to get fixed. cvs2svn is claiming two of the ,v files
to be invalid but to my eyes they look ok.
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: [PATCH] add NO_PERL_XS for environments which are not able to support perl extensions
From: Alex Riesen @ 2006-08-04 14:02 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <81b0412b0608040640s44c0d84et94871bce0271b047@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 626 bytes --]
On 8/4/06, Alex Riesen <raa.lkml@gmail.com> wrote:
> At the moment, the only known example of such environment is Cygwin with
> ActiveState Perl: Makefile, generated by the MakeMaker from ActiveState perl
> distribution is not usable by cygwin's gmake.
>
Damn. Please add attached patch on top of the previous one. I broke
the normal, xs-supported, platforms.
diff --git a/Makefile b/Makefile
index 9cfd677..5d58eca 100644
--- a/Makefile
+++ b/Makefile
@@ -591,7 +591,9 @@ all:
$(MAKE) -C templates
ifndef NO_PERL_XS
-all: perl/Makefile
+.PHONY: perl-dir
+all: perl-dir
+perl-dir: perl/Makefile
$(MAKE) -C perl
endif
[-- Attachment #2: 0001-fix-dependencies-for-xs-supported-platforms.txt --]
[-- Type: text/plain, Size: 628 bytes --]
From ff9c9341341d88b1d3ef55141cab14db45d55cf7 Mon Sep 17 00:00:00 2001
From: Alex Riesen <raa.lkml@gmail.com>
Date: Fri, 4 Aug 2006 15:59:11 +0200
Subject: [PATCH] fix dependencies for xs-supported platforms
Signed-off-by: Alex Riesen <ariesen@harmanbecker.com>
---
Makefile | 4 +++-
1 files changed, 3 insertions(+), 1 deletions(-)
diff --git a/Makefile b/Makefile
index 9cfd677..5d58eca 100644
--- a/Makefile
+++ b/Makefile
@@ -591,7 +591,9 @@ all:
$(MAKE) -C templates
ifndef NO_PERL_XS
-all: perl/Makefile
+.PHONY: perl-dir
+all: perl-dir
+perl-dir: perl/Makefile
$(MAKE) -C perl
endif
--
1.4.2.rc2.g6534
^ permalink raw reply related
* [PATCH] add NO_PERL_XS for environments which are not able to support perl extensions
From: Alex Riesen @ 2006-08-04 13:40 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
[-- Attachment #1: Type: text/plain, Size: 357 bytes --]
At the moment, the only known example of such environment is Cygwin with
ActiveState Perl: Makefile, generated by the MakeMaker from ActiveState perl
distribution is not usable by cygwin's gmake.
Signed-off-by: Alex Riesen <raa.lkml@gmail.com>
---
Makefile | 39 +++++++++++++++++++++++++++++++++------
1 files changed, 33 insertions(+), 6 deletions(-)
[-- Attachment #2: 0001-add-NO_PERL_XS-for-environments-which-are-not-able-to-support-perl-extensions.txt --]
[-- Type: text/plain, Size: 4262 bytes --]
From 259d3ab5925bdf42a2014a7dbf177de7b3ba9dc9 Mon Sep 17 00:00:00 2001
From: Alex Riesen <raa.lkml@gmail.com>
Date: Fri, 4 Aug 2006 15:31:58 +0200
Subject: [PATCH] add NO_PERL_XS for environments which are not able to support perl extensions
At the moment, the only known example of such environment is Cygwin with
ActiveState Perl: Makefile, generated by the MakeMaker from ActiveState perl
distribution is not usable by cygwin's gmake.
Signed-off-by: Alex Riesen <raa.lkml@gmail.com>
---
Makefile | 39 +++++++++++++++++++++++++++++++++------
1 files changed, 33 insertions(+), 6 deletions(-)
diff --git a/Makefile b/Makefile
index d662bd6..9cfd677 100644
--- a/Makefile
+++ b/Makefile
@@ -173,16 +173,19 @@ SCRIPT_SH = \
SCRIPT_PERL = \
git-archimport.perl git-cvsimport.perl git-relink.perl \
- git-shortlog.perl git-rerere.perl \
- git-annotate.perl git-cvsserver.perl \
- git-svnimport.perl git-cvsexportcommit.perl \
- git-send-email.perl git-svn.perl
+ git-shortlog.perl git-rerere.perl git-cvsserver.perl \
+ git-svnimport.perl git-cvsexportcommit.perl git-svn.perl
+
+SCRIPT_XS_PERL = \
+ git-annotate.perl \
+ git-send-email.perl
SCRIPT_PYTHON = \
git-merge-recursive.py
SCRIPTS = $(patsubst %.sh,%,$(SCRIPT_SH)) \
$(patsubst %.perl,%,$(SCRIPT_PERL)) \
+ $(patsubst %.perl,%,$(SCRIPT_XS_PERL)) \
$(patsubst %.py,%,$(SCRIPT_PYTHON)) \
git-cherry-pick git-status git-instaweb
@@ -550,6 +553,9 @@ endif
ifdef NO_ACCURATE_DIFF
BASIC_CFLAGS += -DNO_ACCURATE_DIFF
endif
+ifdef NO_PERL_XS
+ SCRIPT_XS_PERL =
+endif
# Shell quote (do not use $(call) to accommodate ancient setups);
@@ -581,9 +587,13 @@ ### Build rules
all: $(ALL_PROGRAMS) $(BUILT_INS) git$X gitk gitweb/gitweb.cgi
+all:
+ $(MAKE) -C templates
+
+ifndef NO_PERL_XS
all: perl/Makefile
$(MAKE) -C perl
- $(MAKE) -C templates
+endif
strip: $(PROGRAMS) git$X
$(STRIP) $(STRIP_OPTS) $(PROGRAMS) git$X
@@ -613,9 +623,18 @@ common-cmds.h: Documentation/git-*.txt
chmod +x $@+
mv $@+ $@
-$(patsubst %.perl,%,$(SCRIPT_PERL)): perl/Makefile
$(patsubst %.perl,%,$(SCRIPT_PERL)): % : %.perl
rm -f $@ $@+
+ sed -e '1s|#!.*perl|#!$(PERL_PATH_SQ)|' \
+ -e 's/@@GIT_VERSION@@/$(GIT_VERSION)/g' \
+ $@.perl >$@+
+ chmod +x $@+
+ mv $@+ $@
+
+ifndef NO_PERL_XS
+$(patsubst %.perl,%,$(SCRIPT_XS_PERL)): perl/Makefile
+$(patsubst %.perl,%,$(SCRIPT_XS_PERL)): % : %.perl
+ rm -f $@ $@+
INSTLIBDIR=`$(MAKE) -C perl -s --no-print-directory instlibdir` && \
sed -e '1{' \
-e ' s|#!.*perl|#!$(PERL_PATH_SQ)|' \
@@ -629,6 +648,7 @@ common-cmds.h: Documentation/git-*.txt
$@.perl >$@+
chmod +x $@+
mv $@+ $@
+endif
$(patsubst %.py,%,$(SCRIPT_PYTHON)) : % : %.py GIT-CFLAGS
rm -f $@ $@+
@@ -681,6 +701,7 @@ # These can record GIT_VERSION
git$X git.spec \
$(patsubst %.sh,%,$(SCRIPT_SH)) \
$(patsubst %.perl,%,$(SCRIPT_PERL)) \
+ $(patsubst %.perl,%,$(SCRIPT_XS_PERL)) \
$(patsubst %.py,%,$(SCRIPT_PYTHON)) \
: GIT-VERSION-FILE
@@ -747,6 +768,7 @@ XDIFF_OBJS=xdiff/xdiffi.o xdiff/xprepare
rm -f $@ && $(AR) rcs $@ $(XDIFF_OBJS)
+ifndef NO_PERL_XS
PERL_DEFINE = $(BASIC_CFLAGS) -DGIT_VERSION='"$(GIT_VERSION)"'
PERL_DEFINE_SQ = $(subst ','\'',$(PERL_DEFINE))
PERL_LIBS = $(BASIC_LDFLAGS) $(EXTLIBS)
@@ -756,6 +778,7 @@ perl/Makefile: perl/Git.pm perl/Makefile
PREFIX='$(prefix_SQ)' \
DEFINE='$(PERL_DEFINE_SQ)' \
LIBS='$(PERL_LIBS_SQ)')
+endif
doc:
$(MAKE) -C Documentation all
@@ -819,7 +842,9 @@ install: all
$(INSTALL) $(ALL_PROGRAMS) '$(DESTDIR_SQ)$(gitexecdir_SQ)'
$(INSTALL) git$X gitk '$(DESTDIR_SQ)$(bindir_SQ)'
$(MAKE) -C templates DESTDIR='$(DESTDIR_SQ)' install
+ifndef NO_PERL_XS
$(MAKE) -C perl install
+endif
$(INSTALL) -d -m755 '$(DESTDIR_SQ)$(GIT_PYTHON_DIR_SQ)'
$(INSTALL) $(PYMODULES) '$(DESTDIR_SQ)$(GIT_PYTHON_DIR_SQ)'
if test 'z$(bindir_SQ)' != 'z$(gitexecdir_SQ)'; \
@@ -890,8 +915,10 @@ clean:
rm -f $(htmldocs).tar.gz $(manpages).tar.gz
rm -f gitweb/gitweb.cgi
$(MAKE) -C Documentation/ clean
+ifndef NO_PERL_XS
[ ! -f perl/Makefile ] || $(MAKE) -C perl/ clean || $(MAKE) -C perl/ clean
rm -f perl/ppport.h perl/Makefile.old
+endif
$(MAKE) -C templates/ clean
$(MAKE) -C t/ clean
rm -f GIT-VERSION-FILE GIT-CFLAGS
--
1.4.2.rc2.g6534
^ permalink raw reply related
* Re: [KORG] kernel.org/git/ showing nothing
From: Erik Mouw @ 2006-08-04 10:59 UTC (permalink / raw)
To: Jeff Garzik; +Cc: ftpadmin, Git Mailing List
In-Reply-To: <44D3208E.8090403@garzik.org>
On Fri, Aug 04, 2006 at 06:25:18AM -0400, Jeff Garzik wrote:
> When I visit http://www.kernel.org/git/ no projects at all are listed.
>
> At least one other person independently noted this, as well.
It depends on which www.kernel.org you're looking at:
erik@arthur:~ > host www.kernel.org
www.kernel.org CNAME zeus-pub.kernel.org
zeus-pub.kernel.org A 204.152.191.37
zeus-pub.kernel.org A 204.152.191.5
erik@arthur:~ > host 204.152.191.37
Name: zeus-pub2.kernel.org
Address: 204.152.191.37
erik@arthur:~ > host 204.152.191.5
Name: zeus-pub1.kernel.org
Address: 204.152.191.5
http://zeus-pub1.kernel.org/git/ gives you all projects, zeus-pub2
doesn't.
Erik
--
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
^ permalink raw reply
* New Now you could grant your wish Delight in
From: Hilda @ 2006-08-04 10:56 UTC (permalink / raw)
To: git
Good Day!
Do you wish to increase your volume by up to 500%?
Look, now it’s possible with this stuff
Worried it won't work?
Come in here: http://helnall.com/gall/gsm/
Come on in and get it all very cheap!
^ permalink raw reply
* Re: [KORG] kernel.org/git/ showing nothing
From: Junio C Hamano @ 2006-08-04 10:56 UTC (permalink / raw)
To: Jeff Garzik; +Cc: git, ftpadmin
In-Reply-To: <44D3208E.8090403@garzik.org>
Jeff Garzik <jeff@garzik.org> writes:
> When I visit http://www.kernel.org/git/ no projects at all are listed.
>
> At least one other person independently noted this, as well.
DNS round robin -- server at .5 is Ok, .37 seems bad.
^ permalink raw reply
* Re: [KORG] kernel.org/git/ showing nothing
From: Matthias Lederhofer @ 2006-08-04 10:45 UTC (permalink / raw)
To: Jeff Garzik; +Cc: git
In-Reply-To: <44D3208E.8090403@garzik.org>
Jeff Garzik <jeff@garzik.org> wrote:
> When I visit http://www.kernel.org/git/ no projects at all are listed.
>
> At least one other person independently noted this, as well.
The git-daemon and gitweb does not work for me too but rsync does.
^ permalink raw reply
* Re: What's in git.git
From: Jakub Narebski @ 2006-08-04 10:27 UTC (permalink / raw)
To: git
In-Reply-To: <7v7j1on71n.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano wrote:
> - Not-universally-liked Git.pm by Pasky with help from Dennis
> Stosberg, Johannes Schindelin, Pavel Roskin and others.
> One drawback is this pretty much makes Perl scripts that use
> Git.pm unusable with ActiveState right now.
It would be nice if when compiling with NO_GIT_XS (or equivalent) defined,
Git.pm used pure Perl implementation. It would be even better if we could
avoid unnecessary code repetition.
I think it would solve (read: paper on the problem) ActiveState problem...
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* [KORG] kernel.org/git/ showing nothing
From: Jeff Garzik @ 2006-08-04 10:25 UTC (permalink / raw)
To: ftpadmin; +Cc: Git Mailing List
When I visit http://www.kernel.org/git/ no projects at all are listed.
At least one other person independently noted this, as well.
Jeff
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox