* Re: Better value for chunk_size when threaded
From: Nicolas Pitre @ 2007-12-07 1:27 UTC (permalink / raw)
To: Jon Smirl; +Cc: Git Mailing List
In-Reply-To: <9e4733910712061558k19fbc864ia1fb7a3431fd2603@mail.gmail.com>
On Thu, 6 Dec 2007, Jon Smirl wrote:
> I tried some various ideas out for chunk_size and the best strategy I
> found was to simply set it to a constant. How does 20,000 work on
> other CPUs?
That depends on the object size. If you have a repo with big objects
but only 1000 of them for example, then the constant doesn't work.
Ideally I'd opt for a value that tend towards around 5 seconds worth of
work per segment, or something like that. Maybe using the actual
objects size could be another way.
> I'd turn on default threaded support with this change. With threads=1
> versus non-threaded there is no appreciable difference in the time.
Would need a way to determine pthreads availability from Makefile.
> Is there an API to ask how many CPUs are in the system? It would be
> nice to default the number of threads equal to the number of CPUs and
> only use pack.threads=X to override.
If there is one besides futzing with /proc/cpuinfo I'd like to know
about it. Bonus points if it is portable.
Nicolas
^ permalink raw reply
* [PATCH/RFC (take 3)] autoconf: Add test for OLD_ICONV (squelching compiler warning)
From: Jakub Narebski @ 2007-12-07 1:27 UTC (permalink / raw)
To: git
Cc: Linus Torvalds, Blake Ramsdell, Wincent Colaiuta, Junio C Hamano,
Pascal Obry, Ramsay Jones, Arjen Laarhoven, Brian Gernhardt,
Jakub Narebski
In-Reply-To: <alpine.LFD.0.9999.0712061628070.13796@woody.linux-foundation.org>
Update configure.ac (and config.mak.in) to keep up with git
development by adding [compile] test whether your library has an old
iconv(), where the second (input buffer pointer) parameter is declared
with type (const char **) (OLD_ICONV).
Test-proposed-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
CC-ed to all parties (in this and previous thread).
I should probably used AC_LANG_PROGRAM like for NO_C99_FORMAT instead
of generating whole programlet^W test program by hand (I hope that for
example I haven't missed some header file which needs to be included);
I have followed example for NO_ICONV / NEEDS_LIBICONV and
NO_DEFLATE_BOUND test. I'm also not sure if I have put this test in
the correct autoconf section, but that is probably matter of taste.
On Fri, 7 Dec 2007, Linus Torvalds wrote:
>
> On Fri, 7 Dec 2007, Jakub Narebski wrote:
>>
>> The problem is that it should be check that tests for compile time
>> _warnings_; my solution was to use '-Werror' flag to make warning into
>> error, and AC_COMPILE_IFELSE, but this might be gcc only solution.
>>
>> Message-ID: <1196895948-25115-1-git-send-email-jnareb@gmail.com>
>> http://permalink.gmane.org/gmane.comp.version-control.git/67209
>>
>> So please send it.
>
> Umm. Why not just make the test be whether the following compiles cleanly?
>
> #include <iconv.h>
>
> extern size_t iconv(iconv_t cd,
> char **inbuf, size_t *inbytesleft,
> char **outbuf, size_t *outbytesleft);
>
> because if the compiler has seen a "const char **inbuf", then it should
> error out with a "conflicting types for ?iconv?" style message..
[...]
> Totally untested. I don't do autoconf.
Thanks for a suggestion. Implemented in the patch below. It should
work correctly, but could those who have old iconv(), or used to have
old iconv() please check if works correctly for them?
By the way, perhaps the previous idea would work using AC_LANG_WERROR
instead of passing adding [temporarily] GCC `-Werror' option to
`CFLAGS', or use parts of its implementation.
- Macro: AC_LANG_WERROR
Normally Autoconf ignores warnings generated by the compiler,
linker, and preprocessor. If this macro is used, warnings will be
treated as fatal errors instead for the current language. This
macro is useful when the results of configuration will be used
where warnings are unacceptable; for instance, if parts of a
program are built with the GCC `-Werror' option. If the whole
program will be built using `-Werror' it is often simpler to put
`-Werror' in the compiler flags (`CFLAGS' etc.).
On Fri, 7 Dec 2007, Blake Ramsdell wrote:
> On Dec 6, 2007 4:41 PM, Blake Ramsdell <blaker@gmail.com> wrote:
>> On Dec 6, 2007 4:30 PM, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>>> Umm. Why not just make the test be whether the following compiles cleanly?
>>>
>>> #include <iconv.h>
>>>
>>> extern size_t iconv(iconv_t cd,
>>> char **inbuf, size_t *inbytesleft,
>>> char **outbuf, size_t *outbytesleft);
>>>
>>> because if the compiler has seen a "const char **inbuf", then it should
>>> error out with a "conflicting types for 'iconv'" style message..
>>
>> Yeah, this is what I did:
>
> My apologies. Your suggestion is completely different, and should work
> without -Werror. Let me try that.
Is something like the patch below what you wanted to try?
config.mak.in | 1 +
configure.ac | 22 ++++++++++++++++++++++
2 files changed, 23 insertions(+), 0 deletions(-)
diff --git a/config.mak.in b/config.mak.in
index 11d256e..7d5df9b 100644
--- a/config.mak.in
+++ b/config.mak.in
@@ -41,4 +41,5 @@ NO_STRTOUMAX=@NO_STRTOUMAX@
NO_SETENV=@NO_SETENV@
NO_MKDTEMP=@NO_MKDTEMP@
NO_ICONV=@NO_ICONV@
+OLD_ICONV=@OLD_ICONV@
NO_DEFLATE_BOUND=@NO_DEFLATE_BOUND@
diff --git a/configure.ac b/configure.ac
index 5f8a15b..86be19a 100644
--- a/configure.ac
+++ b/configure.ac
@@ -212,6 +212,28 @@ test -n "$NEEDS_SOCKET" && LIBS="$LIBS -lsocket"
## Checks for header files.
+AC_MSG_NOTICE([CHECKS for header files])
+#
+# Define OLD_ICONV if your library has an old iconv(), where the second
+# (input buffer pointer) parameter is declared with type (const char **).
+AC_DEFUN([OLDICONVTEST_SRC], [[
+#include <iconv.h>
+
+extern size_t iconv(iconv_t cd,
+ char **inbuf, size_t *inbytesleft,
+ char **outbuf, size_t *outbytesleft);
+
+int main(void)
+{
+ return 0;
+}
+]])
+AC_MSG_CHECKING([for old iconv()])
+AC_COMPILE_IFELSE(OLDICONVTEST_SRC,
+ [AC_MSG_RESULT([no])],
+ [AC_MSG_RESULT([yes])
+ OLD_ICONV=UnfortunatelyYes])
+AC_SUBST(OLD_ICONV)
## Checks for typedefs, structures, and compiler characteristics.
--
1.5.3.7
^ permalink raw reply related
* Re: [PATCH] Silence iconv warnings on Leopard
From: Blake Ramsdell @ 2007-12-07 0:44 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Jakub Narebski, Wincent Colaiuta, git
In-Reply-To: <985966520712061641y4eba054dj5dea1d0308d50b12@mail.gmail.com>
On Dec 6, 2007 4:41 PM, Blake Ramsdell <blaker@gmail.com> wrote:
> On Dec 6, 2007 4:30 PM, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> > Umm. Why not just make the test be whether the following compiles cleanly?
> >
> > #include <iconv.h>
> >
> > extern size_t iconv(iconv_t cd,
> > char **inbuf, size_t *inbytesleft,
> > char **outbuf, size_t *outbytesleft);
> >
> > because if the compiler has seen a "const char **inbuf", then it should
> > error out with a "conflicting types for 'iconv'" style message..
>
> Yeah, this is what I did:
My apologies. Your suggestion is completely different, and should work
without -Werror. Let me try that.
Blake
--
Blake Ramsdell | http://www.blakeramsdell.com
^ permalink raw reply
* Re: [PATCH] Silence iconv warnings on Leopard
From: Blake Ramsdell @ 2007-12-07 0:41 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Jakub Narebski, Wincent Colaiuta, git
In-Reply-To: <alpine.LFD.0.9999.0712061628070.13796@woody.linux-foundation.org>
On Dec 6, 2007 4:30 PM, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> Umm. Why not just make the test be whether the following compiles cleanly?
>
> #include <iconv.h>
>
> extern size_t iconv(iconv_t cd,
> char **inbuf, size_t *inbytesleft,
> char **outbuf, size_t *outbytesleft);
>
> because if the compiler has seen a "const char **inbuf", then it should
> error out with a "conflicting types for 'iconv'" style message..
Yeah, this is what I did:
diff --git a/configure.ac b/configure.ac
index 5f8a15b..675d3e0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -182,6 +182,29 @@ AC_SUBST(NEEDS_LIBICONV)
AC_SUBST(NO_ICONV)
test -n "$NEEDS_LIBICONV" && LIBS="$LIBS -liconv"
#
+# Define OLD_ICONV if the iconv function prototype uses const** (Darwin and
+# some FreeBSD installations).
+AC_DEFUN([OLDICONVTEST_SRC], [
+#include <iconv.h>
+
+int main(void)
+{
+ char* value = "test";
+
+ (void) iconv (NULL, &value, NULL, NULL, NULL);
+}
+])
+AC_MSG_CHECKING([for old iconv])
+old_CFLAGS="$CFLAGS"
+CFLAGS="$CFLAGS -Werror"
+AC_COMPILE_IFELSE(OLDICONVTEST_SRC,
+ [AC_MSG_RESULT([no])
+ OLD_ICONV=],
+ [AC_MSG_RESULT([yes])
+ OLD_ICONV=UnfortunatelyYes])
+CFLAGS="$old_CFLAGS"
+AC_SUBST(OLD_ICONV)
+#
# Define NO_DEFLATE_BOUND if deflateBound is missing from zlib.
AC_DEFUN([ZLIBTEST_SRC], [
#include <zlib.h>
The problem is that AC_COMPILE_IFELSE doesn't barf on warnings, so I
had to put in the CFLAGS hack to do -Werror (this is what Jakub did
also, I think).
So if this isn't rude to use -Werror (which is probably gcc-specific
in one or more ways), then fine. If it is rude to use -Werror, then
yeah, there needs to be some check for the warning, which I confess in
my five minutes of learning autoconf I don't understand well enough to
say if it's possible.
Blake
--
Blake Ramsdell | http://www.blakeramsdell.com
^ permalink raw reply related
* Re: [PATCH] Silence iconv warnings on Leopard
From: Linus Torvalds @ 2007-12-07 0:30 UTC (permalink / raw)
To: Jakub Narebski; +Cc: Blake Ramsdell, Wincent Colaiuta, git
In-Reply-To: <200712070111.23283.jnareb@gmail.com>
On Fri, 7 Dec 2007, Jakub Narebski wrote:
>
> The problem is that it should be check that tests for compile time
> _warnings_; my solution was to use '-Werror' flag to make warning into
> error, and AC_COMPILE_IFELSE, but this might be gcc only solution.
>
> Message-ID: <1196895948-25115-1-git-send-email-jnareb@gmail.com>
> http://permalink.gmane.org/gmane.comp.version-control.git/67209
>
> So please send it.
Umm. Why not just make the test be whether the following compiles cleanly?
#include <iconv.h>
extern size_t iconv(iconv_t cd,
char **inbuf, size_t *inbytesleft,
char **outbuf, size_t *outbytesleft);
because if the compiler has seen a "const char **inbuf", then it should
error out with a "conflicting types for ‘iconv’" style message..
Just do
$CC -c test-iconv.c
or something.
Totally untested. I don't do autoconf.
Linus
^ permalink raw reply
* Re: Git and GCC
From: Jakub Narebski @ 2007-12-07 0:29 UTC (permalink / raw)
To: Linus Torvalds
Cc: Jon Loeliger, Daniel Berlin, David Miller, Ismail Donmez, gcc,
git
In-Reply-To: <alpine.LFD.0.9999.0712061118050.13796@woody.linux-foundation.org>
Linus Torvalds <torvalds@linux-foundation.org> writes:
> On Thu, 6 Dec 2007, Jon Loeliger wrote:
>> I guess one question I posit is, would it be more accurate
>> to think of this as a "delta net" in a weighted graph rather
>> than a "delta chain"?
>
> It's certainly not a simple chain, it's more of a set of acyclic directed
> graphs in the object list. And yes, it's weigted by the size of the delta
> between objects, and the optimization problem is kind of akin to finding
> the smallest spanning tree (well, forest - since you do *not* want to
> create one large graph, you also want to make the individual trees shallow
> enough that you don't have excessive delta depth).
>
> There are good algorithms for finding minimum spanning trees, but this one
> is complicated by the fact that the biggest cost (by far!) is the
> calculation of the weights itself. So rather than really worry about
> finding the minimal tree/forest, the code needs to worry about not having
> to even calculate all the weights!
>
> (That, btw, is a common theme. A lot of git is about traversing graphs,
> like the revision graph. And most of the trivial graph problems all assume
> that you have the whole graph, but since the "whole graph" is the whole
> history of the repository, those algorithms are totally worthless, since
> they are fundamentally much too expensive - if we have to generate the
> whole history, we're already screwed for a big project. So things like
> revision graph calculation, the main performance issue is to avoid having
> to even *look* at parts of the graph that we don't need to see!)
Hmmm...
I think that these two problems (find minimal spanning forest with
limited depth and traverse graph) with the additional constraint to
avoid calculating weights / avoid calculating whole graph would be
a good problem to present at CompSci course.
Just a thought...
--
Jakub Narebski
Poland
ShadeHawk on #git
^ permalink raw reply
* Re: [PATCH] Silence iconv warnings on Leopard
From: Blake Ramsdell @ 2007-12-07 0:12 UTC (permalink / raw)
To: Jakub Narebski; +Cc: Wincent Colaiuta, git
In-Reply-To: <200712070111.23283.jnareb@gmail.com>
On Dec 6, 2007 4:11 PM, Jakub Narebski <jnareb@gmail.com> wrote:
> I would be interested, as I tried to make a patch to configure.ac which
> does that. The problem is that it should be check that tests for compile
> time _warnings_; my solution was to use '-Werror' flag to make warning
> into error, and AC_COMPILE_IFELSE, but this might be gcc only solution.
That's precisely what I did. If this isn't suitable, then bummer --
impossible ;).
Blake
--
Blake Ramsdell | http://www.blakeramsdell.com
^ permalink raw reply
* Re: [PATCH] Silence iconv warnings on Leopard
From: Jakub Narebski @ 2007-12-07 0:11 UTC (permalink / raw)
To: Blake Ramsdell; +Cc: Wincent Colaiuta, git
In-Reply-To: <985966520712061504s686395d6jf680363c7b3b9de7@mail.gmail.com>
On Fri, 7 Dec 2007, Blake Ramsdell wrote:
> On Dec 6, 2007 11:07 AM, Wincent Colaiuta <win@wincent.com> wrote:
>> Apple ships a newer version of iconv with Leopard (Mac OS X 10.5/Darwin
>> 9). Ensure that OLD_ICONV is not set on any version of Darwin in the
>> 9.x series; this should be good for at least a couple of years, when
>> Darwin 10 comes out and we can invert the sense of the test to
>> specifically check for Darwin 7 or 8.
>> A more sophisticated and robust check is possible for those who use
>> autoconf, but not everybody does that.
>
> I did make a patch for configure.ac that does this. If it's
> interesting, I'll send it along.
I would be interested, as I tried to make a patch to configure.ac which
does that. The problem is that it should be check that tests for compile
time _warnings_; my solution was to use '-Werror' flag to make warning
into error, and AC_COMPILE_IFELSE, but this might be gcc only solution.
Message-ID: <1196895948-25115-1-git-send-email-jnareb@gmail.com>
http://permalink.gmane.org/gmane.comp.version-control.git/67209
So please send it.
Thanks in advance
--
Jakub Narebski
Poland
^ permalink raw reply
* Better value for chunk_size when threaded
From: Jon Smirl @ 2007-12-06 23:58 UTC (permalink / raw)
To: Git Mailing List, Nicolas Pitre
I tried some various ideas out for chunk_size and the best strategy I
found was to simply set it to a constant. How does 20,000 work on
other CPUs?
I'd turn on default threaded support with this change. With threads=1
versus non-threaded there is no appreciable difference in the time.
Is there an API to ask how many CPUs are in the system? It would be
nice to default the number of threads equal to the number of CPUs and
only use pack.threads=X to override.
Making all of this work by default should help when outside people
decide to do a massive import.
diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index 4f44658..4d73be8 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -1645,7 +1645,7 @@ static void ll_find_deltas(struct object_entry
**list, unsigned list_size,
}
/* this should be auto-tuned somehow */
- chunk_size = window * 1000;
+ chunk_size = 20000;
do {
unsigned sublist_size = chunk_size;
with chunk_size = 20000, everything is on a q6600 4GB
threads = 5
time git repack -a -d -f --depth=250 --window=250
real 6m20.123s
user 20m25.841s
sys 0m5.520s
threads = 4
time git repack -a -d -f --depth=250 --window=250
real 6m15.525s
user 20m20.852s
sys 0m5.356s
threads = 4
time git repack -a -d -f
real 1m31.537s
user 3m2.063s
sys 0m3.064s
threads = 1
time git repack -a -d -f --depth=250 --window=250
real 18m46.005s
user 18m43.122s
sys 0m1.228s
threads = 1
time git repack -a -d -f
real 2m57.774s
user 2m54.211s
sys 0m1.228s
Non-threaded
time git repack -a -d -f --depth=250 --window=250
real 18m51.183s
user 18m46.538s
sys 0m1.604s
Non-threaded
time git repack -a -d -f
real 2m54.849s
user 2m51.267s
sys 0m1.412s
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply related
* Re: [RFC] Introduce .git/BRANCH to point to the current branch
From: Robin Rosenberg @ 2007-12-06 23:39 UTC (permalink / raw)
To: Junio C Hamano
Cc: Jakub Narebski, Matthieu Moy, Salikh Zakirov, Git Mailing List
In-Reply-To: <7vir3e428i.fsf@gitster.siamese.dyndns.org>
tisdag 04 december 2007 skrev Junio C Hamano:
> Jakub Narebski <jnareb@gmail.com> writes:
>
> >> Currently, I can do:
> >>
> >> # Oh, what did this look like two commits ago?
> >> $ git checkout HEAD^^
> >> # Ah, OK, let's go back to the tip
> >> $ git checkout branch-name
> >> ^^^^^^^^^^^
> >> But I have to remember and re-type the branch name.
> >
> > No, you don't have. You can use
> > $ git checkout ORIG_HEAD
> > or
> > $ git checkout HEAD@{1}
>
> But the point is he wants to go back to the branch he came from. He
> does not want to detach HEAD at the original commit.
>
> Having said that, I am not sympathetic to "I have to remember".
I abuse git bisect for this temporary switcing. It only gives me a one
level memory, but otoh the git prompt tells me I'm on a discourse.
[me@lathund GIT (rr/abspath|BISECTING)]$ git checkout master
Switched to branch "master"
[me@lathund GIT (master|BISECTING)]$ git checkout HEAD~2
Note: moving to "HEAD~2" which isn't a local branch
If you want to create a new branch from this checkout, you may do so
(now or later) by using -b with the checkout command again. Example:
git checkout -b <new_branch_name>
HEAD is now at afcc4f7... Merge branch 'js/prune-expire'
[me@lathund GIT (afcc4f7...|BISECTING)]$ git bisect reset
Previous HEAD position was afcc4f7... Merge branch 'js/prune-expire'
Switched to branch "rr/abspath"
[me@lathund GIT (rr/abspath)]$
-- robin
^ permalink raw reply
* [PATCH] Change from using email.com to example.com as example domain, as per RFC 2606.
From: David Symonds @ 2007-12-06 23:36 UTC (permalink / raw)
To: Junio Hamano; +Cc: git, David Symonds
Signed-off-by: David Symonds <dsymonds@gmail.com>
---
Documentation/SubmittingPatches | 2 +-
ident.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/Documentation/SubmittingPatches b/Documentation/SubmittingPatches
index 83bf54c..d234c8d 100644
--- a/Documentation/SubmittingPatches
+++ b/Documentation/SubmittingPatches
@@ -10,7 +10,7 @@ Checklist (and a short version for the impatient):
- the first line of the commit message should be a short
description and should skip the full stop
- if you want your work included in git.git, add a
- "Signed-off-by: Your Name <your@email.com>" line to the
+ "Signed-off-by: Your Name <your@example.com>" line to the
commit message (or just use the option "-s" when
committing) to confirm that you agree to the Developer's
Certificate of Origin
diff --git a/ident.c b/ident.c
index 07b4998..7631698 100644
--- a/ident.c
+++ b/ident.c
@@ -175,7 +175,7 @@ static const char *env_hint =
"\n"
"Run\n"
"\n"
-" git config --global user.email \"you@email.com\"\n"
+" git config --global user.email \"you@example.com\"\n"
" git config --global user.name \"Your Name\"\n"
"\n"
"to set your account\'s default identity.\n"
--
1.5.3.1
^ permalink raw reply related
* Re: [PATCH] Silence iconv warnings on Leopard
From: Blake Ramsdell @ 2007-12-06 23:04 UTC (permalink / raw)
To: Wincent Colaiuta; +Cc: git, jnareb
In-Reply-To: <1196968023-45284-1-git-send-email-win@wincent.com>
On Dec 6, 2007 11:07 AM, Wincent Colaiuta <win@wincent.com> wrote:
> Apple ships a newer version of iconv with Leopard (Mac OS X 10.5/Darwin
> 9). Ensure that OLD_ICONV is not set on any version of Darwin in the
> 9.x series; this should be good for at least a couple of years, when
> Darwin 10 comes out and we can invert the sense of the test to
> specifically check for Darwin 7 or 8.
This approach seems fine to me, there was some concern about matching
the OS type / version in the past, but I haven't really seen a better
answer.
> A more sophisticated and robust check is possible for those who use
> autoconf, but not everybody does that.
I did make a patch for configure.ac that does this. If it's
interesting, I'll send it along.
Blake
--
Blake Ramsdell | http://www.blakeramsdell.com
^ permalink raw reply
* Gaahh!!! We've been throough this before
From: A Large Angry SCM @ 2007-12-06 22:47 UTC (permalink / raw)
To: git; +Cc: gitster
Git does not NEED a name or email address to checkout something. So how
are "Big Scary Warnings"(tm) about things that are not required
considered anything but user hostile?
Oh, and "email.com" actually belongs to someone so should not be used as
an example domain name.
"Big Scary Warning"(tm) follows:
git> git-checkout master
*** Your name cannot be determined from your system services (gecos).
Run
git config --global user.email "you@email.com"
git config --global user.name "Your Name"
to set your account's default identity.
Omit --global to set the identity only in this repository.
fatal: empty ident <unknown@localhost> not allowed
git>
^ permalink raw reply
* Re: Git and GCC
From: Jon Smirl @ 2007-12-06 22:44 UTC (permalink / raw)
To: Nicolas Pitre
Cc: Linus Torvalds, Jeff King, Daniel Berlin, Harvey Harrison,
David Miller, ismail, gcc, git
In-Reply-To: <alpine.LFD.0.99999.0712061726240.555@xanadu.home>
On 12/6/07, Nicolas Pitre <nico@cam.org> wrote:
> > > Well, that's possible with a window 25 times larger than the default.
> >
> > Why did it never use more than three cores?
>
> You have 648366 objects total, and only 647457 of them are subject to
> delta compression.
>
> With a window size of 250 and a default thread segment of window * 1000
> that means only 3 segments will be distributed to threads, hence only 3
> threads with work to do.
One little tweak and the clock time drops from 9.5 to 6 minutes. The
tweak makes all four cores work.
jonsmirl@terra:/home/apps/git$ git diff
diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index 4f44658..e0dd12e 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -1645,7 +1645,7 @@ static void ll_find_deltas(struct object_entry
**list, unsigned list_size,
}
/* this should be auto-tuned somehow */
- chunk_size = window * 1000;
+ chunk_size = window * 50;
do {
unsigned sublist_size = chunk_size;
jonsmirl@terra:/home/linux/.git$ time git repack -a -d -f --depth=250
--window=250
Counting objects: 648366, done.
Compressing objects: 100% (647457/647457), done.
Writing objects: 100% (648366/648366), done.
Total 648366 (delta 539043), reused 0 (delta 0)
real 6m2.109s
user 20m0.491s
sys 0m4.608s
jonsmirl@terra:/home/linux/.git$
>
>
> Nicolas
>
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply related
* [OT] Re: Git and GCC
From: Randy Dunlap @ 2007-12-06 22:38 UTC (permalink / raw)
To: David Kastrup
Cc: Junio C Hamano, Jon Loeliger, Linus Torvalds, Daniel Berlin,
David Miller, ismail, gcc, Git List
In-Reply-To: <85r6hzo3y8.fsf@lola.goethe.zz>
On Thu, 06 Dec 2007 23:26:07 +0100 David Kastrup wrote:
> Junio C Hamano <gitster@pobox.com> writes:
>
> > Junio C Hamano <gitster@pobox.com> writes:
> >
> >> Jon Loeliger <jdl@freescale.com> writes:
> >>
> >>> I'd like to learn more about that. Can someone point me to
> >>> either more documentation on it? In the absence of that,
> >>> perhaps a pointer to the source code that implements it?
> >>
> >> See Documentation/technical/pack-heuristics.txt,
> >
> > A somewhat funny thing about this is ...
> >
> > $ git show --stat --summary b116b297
> > commit b116b297a80b54632256eb89dd22ea2b140de622
> > Author: Jon Loeliger <jdl@jdl.com>
> > Date: Thu Mar 2 19:19:29 2006 -0600
> >
> > Added Packing Heursitics IRC writeup.
>
> Ah, fishing for compliments. The cookie baking season...
Indeed. Here are some really good & sweet recipes (IMHO).
http://www.xenotime.net/linux/recipes/
---
~Randy
Features and documentation: http://lwn.net/Articles/260136/
^ permalink raw reply
* Re: Git and GCC
From: Nicolas Pitre @ 2007-12-06 22:30 UTC (permalink / raw)
To: Jon Smirl
Cc: Linus Torvalds, Jeff King, Daniel Berlin, Harvey Harrison,
David Miller, ismail, gcc, git
In-Reply-To: <9e4733910712061422w139273c0gf3cfb04c6ba8c509@mail.gmail.com>
On Thu, 6 Dec 2007, Jon Smirl wrote:
> On 12/6/07, Nicolas Pitre <nico@cam.org> wrote:
> > On Thu, 6 Dec 2007, Jon Smirl wrote:
> >
> > > On 12/6/07, Nicolas Pitre <nico@cam.org> wrote:
> > > > > When I lasted looked at the code, the problem was in evenly dividing
> > > > > the work. I was using a four core machine and most of the time one
> > > > > core would end up with 3-5x the work of the lightest loaded core.
> > > > > Setting pack.threads up to 20 fixed the problem. With a high number of
> > > > > threads I was able to get a 4hr pack to finished in something like
> > > > > 1:15.
> > > >
> > > > But as far as I know you didn't try my latest incarnation which has been
> > > > available in Git's master branch for a few months already.
> > >
> > > I've deleted all my giant packs. Using the kernel pack:
> > > 4GB Q6600
> > >
> > > Using the current thread pack code I get these results.
> > >
> > > The interesting case is the last one. I set it to 15 threads and
> > > monitored with 'top'.
> > > For 0-60% compression I was at 300% CPU, 60-74% was 200% CPU and
> > > 74-100% was 100% CPU. It never used all for cores. The only other
> > > things running were top and my desktop. This is the same load
> > > balancing problem I observed earlier.
> >
> > Well, that's possible with a window 25 times larger than the default.
>
> Why did it never use more than three cores?
You have 648366 objects total, and only 647457 of them are subject to
delta compression.
With a window size of 250 and a default thread segment of window * 1000
that means only 3 segments will be distributed to threads, hence only 3
threads with work to do.
Nicolas
^ permalink raw reply
* Re: Git and GCC
From: David Kastrup @ 2007-12-06 22:26 UTC (permalink / raw)
To: Junio C Hamano
Cc: Jon Loeliger, Linus Torvalds, Daniel Berlin, David Miller, ismail,
gcc, Git List
In-Reply-To: <7vabonczad.fsf@gitster.siamese.dyndns.org>
Junio C Hamano <gitster@pobox.com> writes:
> Junio C Hamano <gitster@pobox.com> writes:
>
>> Jon Loeliger <jdl@freescale.com> writes:
>>
>>> I'd like to learn more about that. Can someone point me to
>>> either more documentation on it? In the absence of that,
>>> perhaps a pointer to the source code that implements it?
>>
>> See Documentation/technical/pack-heuristics.txt,
>
> A somewhat funny thing about this is ...
>
> $ git show --stat --summary b116b297
> commit b116b297a80b54632256eb89dd22ea2b140de622
> Author: Jon Loeliger <jdl@jdl.com>
> Date: Thu Mar 2 19:19:29 2006 -0600
>
> Added Packing Heursitics IRC writeup.
Ah, fishing for compliments. The cookie baking season...
--
David Kastrup, Kriemhildstr. 15, 44793 Bochum
^ permalink raw reply
* Re: Git and GCC
From: Jon Smirl @ 2007-12-06 22:22 UTC (permalink / raw)
To: Nicolas Pitre
Cc: Linus Torvalds, Jeff King, Daniel Berlin, Harvey Harrison,
David Miller, ismail, gcc, git
In-Reply-To: <alpine.LFD.0.99999.0712061645120.555@xanadu.home>
On 12/6/07, Nicolas Pitre <nico@cam.org> wrote:
> On Thu, 6 Dec 2007, Jon Smirl wrote:
>
> > On 12/6/07, Nicolas Pitre <nico@cam.org> wrote:
> > > > When I lasted looked at the code, the problem was in evenly dividing
> > > > the work. I was using a four core machine and most of the time one
> > > > core would end up with 3-5x the work of the lightest loaded core.
> > > > Setting pack.threads up to 20 fixed the problem. With a high number of
> > > > threads I was able to get a 4hr pack to finished in something like
> > > > 1:15.
> > >
> > > But as far as I know you didn't try my latest incarnation which has been
> > > available in Git's master branch for a few months already.
> >
> > I've deleted all my giant packs. Using the kernel pack:
> > 4GB Q6600
> >
> > Using the current thread pack code I get these results.
> >
> > The interesting case is the last one. I set it to 15 threads and
> > monitored with 'top'.
> > For 0-60% compression I was at 300% CPU, 60-74% was 200% CPU and
> > 74-100% was 100% CPU. It never used all for cores. The only other
> > things running were top and my desktop. This is the same load
> > balancing problem I observed earlier.
>
> Well, that's possible with a window 25 times larger than the default.
Why did it never use more than three cores?
>
> The load balancing is solved with a master thread serving relatively
> small object list segments to any work thread that finished with its
> previous segment. But the size for those segments is currently fixed to
> window * 1000 which is way too large when window == 250.
>
> I have to find a way to auto-tune that segment size somehow.
>
> But with the default window size there should not be any such noticeable
> load balancing problem.
>
> Note that threading only happens in the compression phase. The count
> and write phase are hardly paralleled.
>
>
> Nicolas
>
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: Git and GCC
From: Jon Smirl @ 2007-12-06 22:11 UTC (permalink / raw)
To: Nicolas Pitre
Cc: Linus Torvalds, Jeff King, Daniel Berlin, Harvey Harrison,
David Miller, ismail, gcc, git
In-Reply-To: <alpine.LFD.0.99999.0712061645120.555@xanadu.home>
On 12/6/07, Nicolas Pitre <nico@cam.org> wrote:
> On Thu, 6 Dec 2007, Jon Smirl wrote:
>
> > On 12/6/07, Nicolas Pitre <nico@cam.org> wrote:
> > > > When I lasted looked at the code, the problem was in evenly dividing
> > > > the work. I was using a four core machine and most of the time one
> > > > core would end up with 3-5x the work of the lightest loaded core.
> > > > Setting pack.threads up to 20 fixed the problem. With a high number of
> > > > threads I was able to get a 4hr pack to finished in something like
> > > > 1:15.
> > >
> > > But as far as I know you didn't try my latest incarnation which has been
> > > available in Git's master branch for a few months already.
> >
> > I've deleted all my giant packs. Using the kernel pack:
> > 4GB Q6600
> >
> > Using the current thread pack code I get these results.
> >
> > The interesting case is the last one. I set it to 15 threads and
> > monitored with 'top'.
> > For 0-60% compression I was at 300% CPU, 60-74% was 200% CPU and
> > 74-100% was 100% CPU. It never used all for cores. The only other
> > things running were top and my desktop. This is the same load
> > balancing problem I observed earlier.
>
> Well, that's possible with a window 25 times larger than the default.
>
> The load balancing is solved with a master thread serving relatively
> small object list segments to any work thread that finished with its
> previous segment. But the size for those segments is currently fixed to
> window * 1000 which is way too large when window == 250.
>
> I have to find a way to auto-tune that segment size somehow.
That would be nice. Threading is most important on the giant
pack/window combinations. The normal case is fast enough that I don't
real notice it. These giant pack/window combos can run 8-10 hours.
>
> But with the default window size there should not be any such noticeable
> load balancing problem.
I only spend 30 seconds in the compression phase without making the
window larger. It's not long enough to really see what is going on.
>
> Note that threading only happens in the compression phase. The count
> and write phase are hardly paralleled.
>
>
> Nicolas
>
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: XML parsing error from gitweb at freedesktop.org
From: Lars Hjemli @ 2007-12-06 22:11 UTC (permalink / raw)
To: Adam Mercer; +Cc: git
In-Reply-To: <799406d60712061334q33d2dba5r5496ba21069a4547@mail.gmail.com>
On Dec 6, 2007 10:34 PM, Adam Mercer <ramercer@gmail.com> wrote:
> as cgit displays the commit OK
>
> <http://cgit.freedesktop.org/xorg/xserver/commit/?h=xorg-server-1.2-apple&id=48e6a75fbdd0fee86e364f02ace83f20b312a2b2>
>
> leads me to think that the problem lies in gitweb. Could this be a
> problem with gitweb?
>
Actually, it's a problem in both cgit and gitweb, but you need to look
a bit harder to find the error in cgit. It seems that gitweb on
freedesktop.org doesn't detect a file rename (due to diff.renamelimit
maybe?), so it shows the full source of the offending files, while in
cgit you'll need to go from the diff to either the old or the new
sourcefile to get the same error:
http://cgit.freedesktop.org/xorg/xserver/tree/hw/darwin/quartz/applewmExt.h?h=xorg-server-1.2-apple&id=141f69dc3d8d6e7d8ff65607f43700ac11243041
http://cgit.freedesktop.org/xorg/xserver/diff/hw/xquartz/applewm.c?h=xorg-server-1.2-apple&id=48e6a75fbdd0fee86e364f02ace83f20b312a2b2
The problem is a number of unencoded ascii char 12.
--
larsh
^ permalink raw reply
* Re: Git and GCC
From: Nicolas Pitre @ 2007-12-06 22:08 UTC (permalink / raw)
To: Jon Smirl
Cc: Linus Torvalds, Jeff King, Daniel Berlin, Harvey Harrison,
David Miller, ismail, gcc, git
In-Reply-To: <9e4733910712061339n3aef023r22e5b73aac120c8a@mail.gmail.com>
On Thu, 6 Dec 2007, Jon Smirl wrote:
> On 12/6/07, Nicolas Pitre <nico@cam.org> wrote:
> > > When I lasted looked at the code, the problem was in evenly dividing
> > > the work. I was using a four core machine and most of the time one
> > > core would end up with 3-5x the work of the lightest loaded core.
> > > Setting pack.threads up to 20 fixed the problem. With a high number of
> > > threads I was able to get a 4hr pack to finished in something like
> > > 1:15.
> >
> > But as far as I know you didn't try my latest incarnation which has been
> > available in Git's master branch for a few months already.
>
> I've deleted all my giant packs. Using the kernel pack:
> 4GB Q6600
>
> Using the current thread pack code I get these results.
>
> The interesting case is the last one. I set it to 15 threads and
> monitored with 'top'.
> For 0-60% compression I was at 300% CPU, 60-74% was 200% CPU and
> 74-100% was 100% CPU. It never used all for cores. The only other
> things running were top and my desktop. This is the same load
> balancing problem I observed earlier.
Well, that's possible with a window 25 times larger than the default.
The load balancing is solved with a master thread serving relatively
small object list segments to any work thread that finished with its
previous segment. But the size for those segments is currently fixed to
window * 1000 which is way too large when window == 250.
I have to find a way to auto-tune that segment size somehow.
But with the default window size there should not be any such noticeable
load balancing problem.
Note that threading only happens in the compression phase. The count
and write phase are hardly paralleled.
Nicolas
^ permalink raw reply
* Re: how to create v2 patch
From: Andreas Ericsson @ 2007-12-06 22:03 UTC (permalink / raw)
To: Tilman Schmidt; +Cc: Mike Hommey, Pascal Obry, git
In-Reply-To: <475855D6.201@imap.cc>
Tilman Schmidt wrote:
> Am 01.12.2007 14:43 schrieb Mike Hommey:
>> On Sat, Dec 01, 2007 at 02:17:39PM +0100, Pascal Obry wrote:
>>> Tilman Schmidt a écrit :
>>>> I have produced a patch, submitted it to LKML, received a few
>>>> comments, committed appropriate changes to my local git tree,
>>>> and now want to submit a revised patch. How do I do that?
>>>> If I just run git-format-patch again, it produces my original
>>>> patch plus a second one containing my updates, but what I need
>>>> is a single new patch replacing the first one.
>>> Can't you merge both of your changes in your local repository? I would
>>> do that with an interactive rebase.
>> Or just git commit --amend when committing.
>
> Hmm. But wouldn't each of these approaches lead to my original
> commit being removed from my git repository? And isn't removing
> commits that have already been published strongly discouraged?
>
The term "published" means different things for different projects.
For the Linux kernel, "published" is when your commit ends up in a
repository that Linus pulls from.
So long as you're getting suggestions to fix up your patch, it's
safe to assume it hasn't been accepted into one of those repos, and
you can safely --amend the offending commit(s).
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
^ permalink raw reply
* Re: git guidance
From: Phillip Susi @ 2007-12-06 21:46 UTC (permalink / raw)
To: Al Boldi; +Cc: Andreas Ericsson, Linus Torvalds, Jing Xue, linux-kernel, git
In-Reply-To: <200712072155.04643.a1426z@gawab.com>
Al Boldi wrote:
> When you read server, don't read it as localized; a server can be
> distributed. What distinguishes a server from an engine is that it has to
> handle a multi-user use-case. How that is implemented, locally or remotely
> or distributed, is another issue.
And again, git handles both use cases, so what's your point?
> As explained before in this thread, replicating the git tree on the client
> still doesn't provide the required transparency.
It has been pointed out to you that it DOES. Either that or nobody else
understands your nebulous use of "transparency" so maybe you should
define it like we've been asking you. Furthermore, the comment you
replied to said nothing about transparency, nor did your comment it was
in reply to; rather it was pointing out the fact that your statement
that the git can not perform version control on the client is patently
false.
>> How is that different from what every SCM, including git, is doing today?
>> The user needs to tell the scm when it's time to take a snapshot of the
>> current state. Git is distributed though, so committing is usually not the
>> same as publishing. Is that lack of a single command to commit and publish
>> what's nagging you? If it's not, I completely fail to see what you're
>> getting at, unless you've only ever looked at repositories without a
>> worktree attached, or you think that git should work like an editor's
>> "undo" functionality, which would be quite insane.
>
> You need to re-read the thread.
Perhaps you should. We have been trying to get you to explain how you
think git isn't "transparent" while at the same time pointing out how we
think it is. You have failed to demonstrate any evidence to back up
your claims, all of which have been shown to be false.
^ permalink raw reply
* Re: Git and GCC
From: Jon Smirl @ 2007-12-06 21:39 UTC (permalink / raw)
To: Nicolas Pitre
Cc: Linus Torvalds, Jeff King, Daniel Berlin, Harvey Harrison,
David Miller, ismail, gcc, git
In-Reply-To: <alpine.LFD.0.99999.0712061403000.555@xanadu.home>
On 12/6/07, Nicolas Pitre <nico@cam.org> wrote:
> > When I lasted looked at the code, the problem was in evenly dividing
> > the work. I was using a four core machine and most of the time one
> > core would end up with 3-5x the work of the lightest loaded core.
> > Setting pack.threads up to 20 fixed the problem. With a high number of
> > threads I was able to get a 4hr pack to finished in something like
> > 1:15.
>
> But as far as I know you didn't try my latest incarnation which has been
> available in Git's master branch for a few months already.
I've deleted all my giant packs. Using the kernel pack:
4GB Q6600
Using the current thread pack code I get these results.
The interesting case is the last one. I set it to 15 threads and
monitored with 'top'.
For 0-60% compression I was at 300% CPU, 60-74% was 200% CPU and
74-100% was 100% CPU. It never used all for cores. The only other
things running were top and my desktop. This is the same load
balancing problem I observed earlier. Much more clock time was spent
in the 2/1 core phases than the 3 core one.
Threaded, threads = 5
jonsmirl@terra:/home/linux$ time git repack -a -d -f
Counting objects: 648366, done.
Compressing objects: 100% (647457/647457), done.
Writing objects: 100% (648366/648366), done.
Total 648366 (delta 528994), reused 0 (delta 0)
real 1m31.395s
user 2m59.239s
sys 0m3.048s
jonsmirl@terra:/home/linux$
12 seconds counting
53 seconds compressing
38 seconds writing
Without threads,
jonsmirl@terra:/home/linux$ time git repack -a -d -f
warning: no threads support, ignoring pack.threads
Counting objects: 648366, done.
Compressing objects: 100% (647457/647457), done.
Writing objects: 100% (648366/648366), done.
Total 648366 (delta 528999), reused 0 (delta 0)
real 2m54.849s
user 2m51.267s
sys 0m1.412s
jonsmirl@terra:/home/linux$
Threaded, threads = 5
jonsmirl@terra:/home/linux$ time git repack -a -d -f --depth=250 --window=250
Counting objects: 648366, done.
Compressing objects: 100% (647457/647457), done.
Writing objects: 100% (648366/648366), done.
Total 648366 (delta 539080), reused 0 (delta 0)
real 9m18.032s
user 19m7.484s
sys 0m3.880s
jonsmirl@terra:/home/linux$
jonsmirl@terra:/home/linux/.git/objects/pack$ ls -l
total 182156
-r--r--r-- 1 jonsmirl jonsmirl 15561848 2007-12-06 16:15
pack-f1f8637d2c68eb1c964ec7c1877196c0c7513412.idx
-r--r--r-- 1 jonsmirl jonsmirl 170768761 2007-12-06 16:15
pack-f1f8637d2c68eb1c964ec7c1877196c0c7513412.pack
jonsmirl@terra:/home/linux/.git/objects/pack$
Non-threaded:
jonsmirl@terra:/home/linux$ time git repack -a -d -f --depth=250 --window=250
warning: no threads support, ignoring pack.threads
Counting objects: 648366, done.
Compressing objects: 100% (647457/647457), done.
Writing objects: 100% (648366/648366), done.
Total 648366 (delta 539080), reused 0 (delta 0)
real 18m51.183s
user 18m46.538s
sys 0m1.604s
jonsmirl@terra:/home/linux$
jonsmirl@terra:/home/linux/.git/objects/pack$ ls -l
total 182156
-r--r--r-- 1 jonsmirl jonsmirl 15561848 2007-12-06 15:33
pack-f1f8637d2c68eb1c964ec7c1877196c0c7513412.idx
-r--r--r-- 1 jonsmirl jonsmirl 170768761 2007-12-06 15:33
pack-f1f8637d2c68eb1c964ec7c1877196c0c7513412.pack
jonsmirl@terra:/home/linux/.git/objects/pack$
Threaded, threads = 15
jonsmirl@terra:/home/linux$ time git repack -a -d -f --depth=250 --window=250
Counting objects: 648366, done.
Compressing objects: 100% (647457/647457), done.
Writing objects: 100% (648366/648366), done.
Total 648366 (delta 539080), reused 0 (delta 0)
real 9m18.325s
user 19m14.340s
sys 0m3.996s
jonsmirl@terra:/home/linux$
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: how to create v2 patch
From: Pascal Obry @ 2007-12-06 21:38 UTC (permalink / raw)
To: Tilman Schmidt; +Cc: Mike Hommey, git
In-Reply-To: <475855D6.201@imap.cc>
Tilman Schmidt a écrit :
> Hmm. But wouldn't each of these approaches lead to my original
> commit being removed from my git repository? And isn't removing
> commits that have already been published strongly discouraged?
They won't be removed, just changed/merged... and that's what you were
looking for or I did not understand your question! This is not bad
practice as it is done on YOUR repository. Of course this should never
be done on a pushed/published changeset.
Pascal.
--
--|------------------------------------------------------
--| Pascal Obry Team-Ada Member
--| 45, rue Gabriel Peri - 78114 Magny Les Hameaux FRANCE
--|------------------------------------------------------
--| http://www.obry.net
--| "The best way to travel is by means of imagination"
--|
--| gpg --keyserver wwwkeys.pgp.net --recv-key C1082595
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox