* cygwin: push/pull takes very long time
@ 2006-03-02 13:26 Alex Riesen
2006-03-02 17:09 ` Alex Riesen
0 siblings, 1 reply; 5+ messages in thread
From: Alex Riesen @ 2006-03-02 13:26 UTC (permalink / raw)
To: Git Mailing List
Hi,
I didn't really notice when it happened, but since some time
push and pull takes *very* long. git-pack-objects appears to
be the major offender:
$ git pull ../src march:march
Generating pack...
Done counting 20238 objects.
Result has 12 objects.
Deltifying 12 objects.
8% (1/12) done
It's still working, half an hour since it was started.
It can be seen that it is constantly growing and shrinking
(for about 20k,50k,80k,100k back and forth).
I did a small instrumentation to figure out how much
and how often is allocated, but it's waiting for the
git-pack-objects to finish. "Counting objects" was
slow too, with git-rev-list "pulsating" in a similar
way (most commonly allocated objects in the size
range from 16 to 256).
I continue looking for the problem (I didn't do bisect yet,
want to wait for mem profile first), but any ideas on how
to fix that are very welcome.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: cygwin: push/pull takes very long time
2006-03-02 13:26 cygwin: push/pull takes very long time Alex Riesen
@ 2006-03-02 17:09 ` Alex Riesen
2006-03-02 21:54 ` Alex Riesen
0 siblings, 1 reply; 5+ messages in thread
From: Alex Riesen @ 2006-03-02 17:09 UTC (permalink / raw)
To: Git Mailing List
On 3/2/06, Alex Riesen <raa.lkml@gmail.com> wrote:
> It's still working, half an hour since it was started.
> It can be seen that it is constantly growing and shrinking
> (for about 20k,50k,80k,100k back and forth).
The distribution is (there are some git-commits and gitks
probably, here as git-read-tree?) below. The first number
after program name is lower range limit, the second - upper limit.
The number after them - counter of *alloc calls.
The strange pair 65536:99999 means very big objects, more than 1Mb.
I'll cleanup the profiling code and send it as well soon
(I had to instrument x*alloc).
git-read-tree:16:256 127002
git-rev-list:16:256 44775
git-read-tree:0:16 22577
git-pack-objects:16:256 12495
git-pack-objects:256:4096 12303
git-pack-objects:4096:65536 6929
git-diff-index:16:256 4381
git-http-fetch:16:256 3535
git-diff-index:16:256 2914
git-write-tree:4096:65536 2876
git-http-fetch:0:16 2111
git-fetch-pack:16:256 1646
git-fetch-pack:0:16 1522
git-rev-list:256:4096 1067
:16:256 976
git-read-tree:256:4096 965
git-http-fetch:256:4096 835
:0:16 825
git-ls-files:0:16 791
git-rev-list:0:16 748
git-fetch-pack:256:4096 698
git-upload-pack:16:256 610
git-pack-objects:65536:1048576 594
git-rev-parse:16:256 571
:256:4096 554
git-rev-parse:0:16 492
git-upload-pack:0:16 469
git-upload-pack:256:4096 248
git-cat-file:0:16 243
git-peek-remote:16:256 242
git-merge-base:0:16 229
git-merge-base:16:256 208
git-cat-file:16:256 167
git-rev-parse:256:4096 141
git-rev-parse:16:256 117
git-rev-parse:0:16 116
git-cat-file:256:4096 80
git-pack-objects:0:16 73
git-diff-index:0:16 65
git-merge-base:256:4096 59
git-pack-objects:65536:99999 54
git-current-branch:0:16 48
git-diff-index:0:16 47
git-diff-tree:16:256 42
git-diff-index:256:4096 36
git-unpack-objects:16:256 31
git-diff-tree:16:256 30
git-ls-files:16:256 27
git-update-index:16:256 27
git-diff-index:256:4096 24
git-write-tree:16:256 23
git-cat-file:16:256 22
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: cygwin: push/pull takes very long time
2006-03-02 17:09 ` Alex Riesen
@ 2006-03-02 21:54 ` Alex Riesen
2006-03-03 0:28 ` Christopher Faylor
0 siblings, 1 reply; 5+ messages in thread
From: Alex Riesen @ 2006-03-02 21:54 UTC (permalink / raw)
To: Git Mailing List
Alex Riesen, Thu, Mar 02, 2006 18:09:23 +0100:
> I'll cleanup the profiling code and send it as well soon
> (I had to instrument x*alloc).
This is not exactly the same. It counts free as well, even if that is
not really interesting - there are places were there is more frees
than allocs. Probably something missed or a result coming from libc.
Also it is _not_ the code I used for windows. I had to have a global
variable for argv[0], which needs modification of all main()s, which
gets too easily out of sync.
BTW, maybe someone has an idea as to how attach valgrind to everything?
(I mean, without changing every script. Maybe modify git.c?)
Anyway, here it is, could be useful for something.
---
diff --git a/Makefile b/Makefile
index a5eb0c4..18fc802 100644
--- a/Makefile
+++ b/Makefile
@@ -206,7 +206,7 @@ LIB_OBJS = \
quote.o read-cache.o refs.o run-command.o \
server-info.o setup.o sha1_file.o sha1_name.o strbuf.o \
tag.o tree.o usage.o config.o environment.o ctype.o copy.o \
- fetch-clone.o \
+ fetch-clone.o alloc.o \
$(DIFF_OBJS)
LIBS = $(LIB_FILE)
diff --git a/alloc.c b/alloc.c
new file mode 100644
index 0000000..76727e6
--- /dev/null
+++ b/alloc.c
@@ -0,0 +1,108 @@
+/* simple allocation logging */
+#include <unistd.h>
+#include <limits.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include "git-compat-util.h"
+
+#undef free
+
+#define SHIFT 2
+static unsigned cnts[12] = {0,0,0,0, 0,0,0,0};
+static unsigned fcnt = 0;
+
+static int exit_hook = 1;
+
+static void alloc_profile()
+{
+ char argv0[PATH_MAX];
+ FILE *fp;
+ fp = fopen("/proc/self/cmdline", "r");
+ if ( !fp ) {
+ sprintf(argv0, "%d", getpid());
+ } else {
+ fread(argv0, sizeof(argv0), 1, fp);
+ fclose(fp);
+ char *s = strrchr(argv0, '/');
+ if ( s )
+ memmove(argv0, s + 1, strlen(s));
+ }
+ fp = fopen("/tmp/git-alloc", "ab");
+ if ( !fp )
+ return;
+ unsigned i, c = 0;
+ for ( i = 0; i < sizeof(cnts)/sizeof(*cnts); ++i ) {
+ if ( !cnts[i] )
+ continue;
+ fprintf(fp, "%s %u %u:%u %u times\n",
+ argv0,
+ i,
+ i ? 1 << i * SHIFT: 0,
+ 1 << (i+1) * SHIFT,
+ cnts[i]);
+ c += cnts[i];
+ }
+ fprintf(fp, "%s alloc-free = %u\n", argv0, c - fcnt);
+ fclose(fp);
+}
+
+static inline void count(size_t size)
+{
+ if ( exit_hook ) {
+ exit_hook = 0;
+ atexit(alloc_profile);
+ }
+ unsigned i = 0;
+ while ( size && i < sizeof(cnts)/sizeof(*cnts) ) {
+ size >>= SHIFT;
+ if ( size )
+ ++i;
+ else {
+ ++cnts[i];
+ return;
+ }
+ }
+ ++cnts[sizeof(cnts)/sizeof(*cnts)-1];
+}
+
+void *xmalloc(size_t size)
+{
+ void *ret = malloc(size);
+ count(size);
+ if (!ret && !size)
+ ret = malloc(1);
+ if (!ret)
+ die("Out of memory, malloc failed, %u bytes requested", size);
+ return ret;
+}
+
+void *xrealloc(void *ptr, size_t size)
+{
+ void *ret = realloc(ptr, size);
+ count(size);
+ if (!ret && !size)
+ ret = realloc(ptr, 1);
+ if (!ret)
+ die("Out of memory, realloc failed, %u bytes requested", size);
+ return ret;
+}
+
+void *xcalloc(size_t nmemb, size_t size)
+{
+ void *ret = calloc(nmemb, size);
+ count(size);
+ if (!ret && (!nmemb || !size))
+ ret = calloc(1, 1);
+ if (!ret)
+ die("Out of memory, calloc failed");
+ return ret;
+}
+
+void xfree(void *ptr)
+{
+ if ( !ptr )
+ return;
+ free(ptr);
+ ++fcnt;
+}
+
diff --git a/git-compat-util.h b/git-compat-util.h
index f982b8e..ab4f855 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -73,35 +73,11 @@ extern void gitunsetenv(const char *);
extern char *gitstrcasestr(const char *haystack, const char *needle);
#endif
-static inline void *xmalloc(size_t size)
-{
- void *ret = malloc(size);
- if (!ret && !size)
- ret = malloc(1);
- if (!ret)
- die("Out of memory, malloc failed");
- return ret;
-}
-
-static inline void *xrealloc(void *ptr, size_t size)
-{
- void *ret = realloc(ptr, size);
- if (!ret && !size)
- ret = realloc(ptr, 1);
- if (!ret)
- die("Out of memory, realloc failed");
- return ret;
-}
-
-static inline void *xcalloc(size_t nmemb, size_t size)
-{
- void *ret = calloc(nmemb, size);
- if (!ret && (!nmemb || !size))
- ret = calloc(1, 1);
- if (!ret)
- die("Out of memory, calloc failed");
- return ret;
-}
+void *xmalloc(size_t size);
+void *xrealloc(void *ptr, size_t size);
+void *xcalloc(size_t nmemb, size_t size);
+void xfree(void *ptr);
+#define free(p) xfree(p)
static inline ssize_t xread(int fd, void *buf, size_t len)
{
diff --git a/prof-decode.sh b/prof-decode.sh
new file mode 100755
index 0000000..9c07e05
--- /dev/null
+++ b/prof-decode.sh
@@ -0,0 +1,25 @@
+#!/bin/bash
+if [ $# = 0 ]; then
+ set -- /tmp/git-alloc
+fi
+cat "$@" | perl -e '
+ while(<>) {
+ if (/(\S*) \d+ (\d+:\d+) (\d+)/) {
+ $c{$1}->{$2} += $3;
+ $c{$1}->{all} += $3
+ }
+ if (/(\S*) alloc-free = (\d+)/) { $d{$1} += $2 }
+ }
+ foreach $k (sort {$c{$b}->{all} <=> $c{$a}->{all}} grep {$c{$_}} keys %c) {
+ for ( sort {
+ if ( $a eq "all" ) { -1 }
+ elsif ( $b eq "all" ) { 1 }
+ else {
+ "$a $b" =~ /(\d+):\d+ (\d+):\d+/;
+ $1 <=> $2 }
+ } grep { $c{$k}->{$_} } keys %{$c{$k}} ) {
+ printf "%20s\t%20s\t%d\n", $k, $_, $c{$k}->{$_}
+ }
+ printf "%20s\t%d leaks\n\n", $k, $d{$k}
+ }
+'
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: cygwin: push/pull takes very long time
2006-03-02 21:54 ` Alex Riesen
@ 2006-03-03 0:28 ` Christopher Faylor
2006-03-03 9:10 ` Alex Riesen
0 siblings, 1 reply; 5+ messages in thread
From: Christopher Faylor @ 2006-03-03 0:28 UTC (permalink / raw)
To: Git Mailing List
On Thu, Mar 02, 2006 at 10:54:08PM +0100, Alex Riesen wrote:
>Alex Riesen, Thu, Mar 02, 2006 18:09:23 +0100:
>>I'll cleanup the profiling code and send it as well soon (I had to
>>instrument x*alloc).
>
>This is not exactly the same. It counts free as well, even if that is
>not really interesting - there are places were there is more frees than
>allocs. Probably something missed or a result coming from libc.
>
>Also it is _not_ the code I used for windows. I had to have a global
>variable for argv[0], which needs modification of all main()s, which
>gets too easily out of sync.
I wasn't following this discussion closely so maybe this is useless
information, but for Cygwin you can either use the undocumented global
__argv or you can use /proc/cmdline. /proc/self/cmdline is going to be
pretty slow, however.
It looks like pure Windows console apps define _argv in stdlib.h also
but I've never used this and don't know if it is what it looks like.
cgf
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: cygwin: push/pull takes very long time
2006-03-03 0:28 ` Christopher Faylor
@ 2006-03-03 9:10 ` Alex Riesen
0 siblings, 0 replies; 5+ messages in thread
From: Alex Riesen @ 2006-03-03 9:10 UTC (permalink / raw)
To: Christopher Faylor; +Cc: Git Mailing List
On 3/3/06, Christopher Faylor <me@cgf.cx> wrote:
> On Thu, Mar 02, 2006 at 10:54:08PM +0100, Alex Riesen wrote:
> >Alex Riesen, Thu, Mar 02, 2006 18:09:23 +0100:
> >>I'll cleanup the profiling code and send it as well soon (I had to
> >>instrument x*alloc).
> >
> >This is not exactly the same. It counts free as well, even if that is
> >not really interesting - there are places were there is more frees than
> >allocs. Probably something missed or a result coming from libc.
> >
> >Also it is _not_ the code I used for windows. I had to have a global
> >variable for argv[0], which needs modification of all main()s, which
> >gets too easily out of sync.
>
> I wasn't following this discussion closely so maybe this is useless
> information, but for Cygwin you can either use the undocumented global
> __argv or you can use /proc/cmdline. /proc/self/cmdline is going to be
> pretty slow, however.
Oh, thanks. The speed is of no problem here: it's in atexit callback.
> It looks like pure Windows console apps define _argv in stdlib.h also
> but I've never used this and don't know if it is what it looks like.
It works as usual argv. I used "char **__argv", got the message from linker:
Info: resolving ___argv by linking to __imp____argv (auto-import)
which I almost expected.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2006-03-03 9:10 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2006-03-02 13:26 cygwin: push/pull takes very long time Alex Riesen
2006-03-02 17:09 ` Alex Riesen
2006-03-02 21:54 ` Alex Riesen
2006-03-03 0:28 ` Christopher Faylor
2006-03-03 9:10 ` Alex Riesen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).