* Re: irc usage..
From: Martin Langhoff @ 2006-05-22 23:15 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Matthias Urlichs, git
In-Reply-To: <7v8xotadm3.fsf@assigned-by-dhcp.cox.net>
On 5/23/06, Junio C Hamano <junkio@cox.net> wrote:
> > I simply was too lazy to count the actual filenames' lengths. ;-)
>
> I think cvsimport predates that option, but these days that loop
> can be optimized by feeding --index-info from standard input.
Oh, yep, that'd be a good addition. I think we can also cut down on
the number of fork+exec calls (as Linus points out they are killing
us) by caching some data we should already have that we are repeatedly
asking from git-ref-parse.
Other TODOs from my reading of the code last night...
- Switch from line-oriented reads to block reads when fetching files
from CVS. This gentoo has repo has some large binary blobs in it and
we end up slurping them into memory.
- Stop abusing globals in commit() -- pass the commit data as parameters.
- Further profiling? Whatever we are doing, we aren't doing it fast :(
Will be trying to do those things in the next few days, don't mind if
someone jumps in as well.
martin
^ permalink raw reply
* Re: Current Issues #3
From: Shawn Pearce @ 2006-05-22 23:12 UTC (permalink / raw)
To: Daniel Barkalow; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.64.0605221738090.6713@iabervon.org>
Daniel Barkalow <barkalow@iabervon.org> wrote:
> On Mon, 22 May 2006, Junio C Hamano wrote:
>
> > * reflog
> >
> > I still haven't merged this series to "next" -- I do not have
> > much against what the code does, but I am unconvinced if it is
> > useful. Also objections raised on the list that this can be
> > replaced by making sure that a repository that has hundreds of
> > tags usable certainly have a point.
>
> I think it would make gitweb's summary view clearer, and Linus seemed
> interested in being able to look up what happened in the fast forward
> which was the first of several merges in a day.
>
> It could be replaced by a repository with hundreds of machine-readable
> tags with code to parse dates into queries for suitable tags. But I don't
> think there's an advantage to using the tag mechanism here, because you
> never want to look the history up by exactly which history it is (the
> thing that a tag ref is good for); you'll be looking for whatever reflog
> item is the newest not after a specified time, where the specified time is
> almost never a time that a reflog item was created.
The thing is this might also be easily represented as a structure
of tags; for example:
refs/logs/heads/<ref>/<year>/<month>/<day> <hour>:<min>:<sec>:<seq>
where the tag is a tag of the commit which was valid in that ref
at that time. Searching for an entry "around a particular time"
isn't that much more difficult than parsing a file, you just have
to walk backwards through the sorted directory listings then read
the tag object which matches; that tag object will point at the
tree/commit/tag which is was in that ref..
What's ugly about this is simply the disk storage: a ref file is an
expensive thing (relatively speaking) on most UNIX file systems due
to the inode overhead. If this was stored in a more compact format
(such as a GIT tree) then this would cost very little.
So the alternative that I have been mentaly kicking around for
the past two days is storing the GIT_DIR/refs directory within a
standard GIT tree. This of course would need to be an option that
gets enabled by the user as currently most tools expect the refs
directory to actually be a directory, not a tree. The advantage here
is that unlike proposed reflog it is a compact ref representation
which could be used by other features, such as tagging a GIT
commit with the unique name of the same change from another SCM.
Or tagging your repository on every automated build, which runs
once every 5 minutes.
--
Shawn.
^ permalink raw reply
* git-diff-tree crashes on ubuntu kernel git repository
From: Torgil Svensson @ 2006-05-22 23:09 UTC (permalink / raw)
To: git
Hi
It seems like git-diff-tree has some problems with moved files:
$ git-diff-tree -p --stat --summary -M
348f179e3195448cea49c98a79cce8c7f446ce26
343ca16424ba031b37e4df49afddaee098a8f347 | wc -l
*** glibc detected *** free(): invalid pointer: 0x12ecbbf0 ***
6101
As can be seen below there is some obvious error in the output just
prior to the crash:
drivers/w1/{masters => }/ds_w1_bridge.c | 38
This file is moved into "w1/masters" by commit
bd529cfb40c427d5b5aae0d315afb9f0a1da5e76
$ git --version
git version 1.3.3.g5e36
$ cat .git/remotes/origin
URL: git://git.kernel.org/pub/scm/linux/kernel/git/bcollins/ubuntu-2.6
Pull: refs/heads/master:refs/heads/origin
$ gdb git-diff-tree
(gdb) run -p --stat --summary -M
348f179e3195448cea49c98a79cce8c7f446ce26
343ca16424ba031b37e4df49afddaee098a8f347
<...lots of files...>
drivers/video/w100fb.c | 162
drivers/video/w100fb.h | 748 -
drivers/w1/Kconfig | 62
drivers/w1/Makefile | 10
drivers/w1/{masters => }/ds_w1_bridge.c | 38
*** glibc detected *** free(): invalid pointer: 0x12ecbbf0 ***
Program received signal SIGABRT, Aborted.
0xffffe410 in __kernel_vsyscall ()
(gdb) bt
#0 0xffffe410 in __kernel_vsyscall ()
#1 0xb7d7e9a1 in raise () from /lib/tls/i686/cmov/libc.so.6
#2 0xb7d802b9 in abort () from /lib/tls/i686/cmov/libc.so.6
#3 0xb7db287a in __fsetlocking () from /lib/tls/i686/cmov/libc.so.6
#4 0xb7db8fd4 in malloc_usable_size () from /lib/tls/i686/cmov/libc.so.6
#5 0xb7db934a in free () from /lib/tls/i686/cmov/libc.so.6
#6 0x08056902 in show_stats (data=0x8deff80) at diff.c:392
#7 0x08058466 in diff_flush (options=0x80686b0) at diff.c:1999
#8 0x0805b143 in log_tree_diff_flush (opt=0x8068680) at log-tree.c:82
#9 0x08049d11 in main (argc=0, argv=0xbfcf8a14) at diff-tree.c:130
(gdb)
As shown above I can easily recreate the crash if you want more info.
Thank you for a wonderful tool.
//Torgil
^ permalink raw reply
* Re: Local clone/fetch with cogito is glacial
From: Petr Baudis @ 2006-05-22 23:08 UTC (permalink / raw)
To: H. Peter Anvin; +Cc: Sean, git
In-Reply-To: <4472432A.8010002@zytor.com>
Dear diary, on Tue, May 23, 2006 at 01:03:06AM CEST, I got a letter
where "H. Peter Anvin" <hpa@zytor.com> said that...
> Petr Baudis wrote:
> >
> >Even rsync and HTTP cg-clones? git:// and git+ssh:// fetching follows an
> >almost entirely different code patch and it's much more efficient since
> >I just accumulate the tag object ids I want to check and then pour them
> >to git-fetch-pack - I cannot do that with git-(local|http)-fetch. :-(
> >
>
> No, but git-fetch-pack could operate over a local pipe just fine (after
> all, all it does is ssh an "git-send-pack" command to the other side.)
Yes, but in that case it couldn't hardlink the objects so you would see
quite a big bump in disk usage if you have many local clones of the same
repo.
That said, hardlinking is probably not all that big an advantage if you
repack often, repack everywhere, and in the many-repositories cases it
might be more sensible to use alternates (which is what cg-clone -l
should really do instead of symlinking), so it might be well worth
the sacrifice.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Right now I am having amnesia and deja-vu at the same time. I think
I have forgotten this before.
^ permalink raw reply
* Re: Local clone/fetch with cogito is glacial
From: H. Peter Anvin @ 2006-05-22 23:03 UTC (permalink / raw)
To: Petr Baudis; +Cc: Sean, git
In-Reply-To: <20060522225054.GL11941@pasky.or.cz>
Petr Baudis wrote:
>
> Even rsync and HTTP cg-clones? git:// and git+ssh:// fetching follows an
> almost entirely different code patch and it's much more efficient since
> I just accumulate the tag object ids I want to check and then pour them
> to git-fetch-pack - I cannot do that with git-(local|http)-fetch. :-(
>
No, but git-fetch-pack could operate over a local pipe just fine (after all, all it does
is ssh an "git-send-pack" command to the other side.)
-hpa
^ permalink raw reply
* Re: Local clone/fetch with cogito is glacial
From: Petr Baudis @ 2006-05-22 22:50 UTC (permalink / raw)
To: H. Peter Anvin; +Cc: Sean, git
In-Reply-To: <447239F0.9030705@zytor.com>
Dear diary, on Tue, May 23, 2006 at 12:23:44AM CEST, I got a letter
where "H. Peter Anvin" <hpa@zytor.com> said that...
> Petr Baudis wrote:
> >git-clone has an advantage here since it clones _everything_ while
> >Cogito fetches only stuff related to the branch you are cloning, and
> >verifying if what it fetches is sensible for you unfortunately takes a
> >lot of time. :/ I guess there is no way to verify presence of multiple
> >objects at once and there is also no way to order local fetch of
> >multiple objects at once.
>
> Note that non-local cg-clones are at least an order of magnitude faster,
> even when the nonlocal is just git+ssh:. One could presumably do the same
> thing over a pipe.
Even rsync and HTTP cg-clones? git:// and git+ssh:// fetching follows an
almost entirely different code patch and it's much more efficient since
I just accumulate the tag object ids I want to check and then pour them
to git-fetch-pack - I cannot do that with git-(local|http)-fetch. :-(
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Right now I am having amnesia and deja-vu at the same time. I think
I have forgotten this before.
^ permalink raw reply
* Re: irc usage..
From: Junio C Hamano @ 2006-05-22 22:39 UTC (permalink / raw)
To: Matthias Urlichs; +Cc: git
In-Reply-To: <20060522214128.GE16677@kiste.smurf.noris.de>
Matthias Urlichs <smurf@smurf.noris.de> writes:
> Hi,
>
> Linus Torvalds:
>> I wonder why those "git-update-index" calls seem to be (assuming I read
>> the perl correctly) done only a few files at a time. We can do a hundreds
>> in one go, but it seems to want to do just ten files or something at the
>> same time.
>
> No, fifty.
>
> I simply was too lazy to count the actual filenames' lengths. ;-)
I think cvsimport predates that option, but these days that loop
can be optimized by feeding --index-info from standard input.
^ permalink raw reply
* Re: Local clone/fetch with cogito is glacial
From: H. Peter Anvin @ 2006-05-22 22:23 UTC (permalink / raw)
To: Petr Baudis; +Cc: Sean, git
In-Reply-To: <20060522220206.GA10488@pasky.or.cz>
Petr Baudis wrote:
>
> What about incremental fetches using git-fetch? From a quick scan of the
> git-fetch automagic tags following code, it seems to be even
> significantly more expensive than Cogito's (in terms of number of
> forks).
>
Well, I haven't used git-fetch, so I can't comment on that one.
> git-clone has an advantage here since it clones _everything_ while
> Cogito fetches only stuff related to the branch you are cloning, and
> verifying if what it fetches is sensible for you unfortunately takes a
> lot of time. :/ I guess there is no way to verify presence of multiple
> objects at once and there is also no way to order local fetch of
> multiple objects at once.
Note that non-local cg-clones are at least an order of magnitude faster, even when the
nonlocal is just git+ssh:. One could presumably do the same thing over a pipe.
-hpa
^ permalink raw reply
* Re: irc usage..
From: Linus Torvalds @ 2006-05-22 22:18 UTC (permalink / raw)
To: Matthias Urlichs
Cc: Martin Langhoff, Donnie Berkholz, Yann Dirson, Git Mailing List,
Johannes Schindelin
In-Reply-To: <20060522214128.GE16677@kiste.smurf.noris.de>
On Mon, 22 May 2006, Matthias Urlichs wrote:
>
> The beast *was* mainly written to do this remotely...
I don't think the remote usability is valid, except for some really small
repositories. The fact that it takes hours even when the CVS server is
local doesn't bode well for doing it remotely for any but the most trivial
things.
I really think it would be better to have local use be the optimized case,
with remote being the "it's _possible_" case.
Linus
^ permalink raw reply
* Re: irc usage..
From: Matthias Urlichs @ 2006-05-22 21:41 UTC (permalink / raw)
To: Linus Torvalds
Cc: Martin Langhoff, Donnie Berkholz, Yann Dirson, Git Mailing List,
Johannes Schindelin
In-Reply-To: <Pine.LNX.4.64.0605221256090.3697@g5.osdl.org>
[-- Attachment #1: Type: text/plain, Size: 872 bytes --]
Hi,
Linus Torvalds:
> I wonder why those "git-update-index" calls seem to be (assuming I read
> the perl correctly) done only a few files at a time. We can do a hundreds
> in one go, but it seems to want to do just ten files or something at the
> same time.
No, fifty.
I simply was too lazy to count the actual filenames' lengths. ;-)
> That thing would probably be an order of magnitude faster if written to
> use the git library interfaces directly. Of course, the CVS part is
> probably a big overhead, so it might not help much
The beast *was* mainly written to do this remotely...
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
- -
The worst form of inequality is to try to make unequal things equal.
-- Aristotle
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 191 bytes --]
^ permalink raw reply
* Re: Current Issues #3
From: Carl Worth @ 2006-05-22 22:02 UTC (permalink / raw)
To: Daniel Barkalow; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.64.0605221738090.6713@iabervon.org>
[-- Attachment #1: Type: text/plain, Size: 214 bytes --]
On Mon, 22 May 2006 17:54:28 -0400 (EDT), Daniel Barkalow wrote:
> On Mon, 22 May 2006, Junio C Hamano wrote:
>
> > * reflog
Am I the only one that read that as re-flog rather than ref-log the
first time?
-Carl
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: Local clone/fetch with cogito is glacial
From: Petr Baudis @ 2006-05-22 22:02 UTC (permalink / raw)
To: H. Peter Anvin; +Cc: Sean, git
In-Reply-To: <44722A8F.9020609@zytor.com>
Dear diary, on Mon, May 22, 2006 at 11:18:07PM CEST, I got a letter
where "H. Peter Anvin" <hpa@zytor.com> said that...
> Sean wrote:
> >On Sun, 21 May 2006 16:47:45 -0700
> >"H. Peter Anvin" <hpa@zytor.com> wrote:
> >
> >>It appears that doing a *local* -- meaning using a file path or file URL
> >>-- clone or fetch with cogito is just glacial when the repository has an
> >>even moderate number of tags (and it's fetching the tags that takes all
> >>the time.) That's a really serious problem for me.
> >>
> >
> >Peter, does git clone work acceptably for you?
> >
>
> Well, it does, except it doesn't set up the cogito branches (which one can
> of course copy manually.)
What about incremental fetches using git-fetch? From a quick scan of the
git-fetch automagic tags following code, it seems to be even
significantly more expensive than Cogito's (in terms of number of
forks).
git-clone has an advantage here since it clones _everything_ while
Cogito fetches only stuff related to the branch you are cloning, and
verifying if what it fetches is sensible for you unfortunately takes a
lot of time. :/ I guess there is no way to verify presence of multiple
objects at once and there is also no way to order local fetch of
multiple objects at once.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Right now I am having amnesia and deja-vu at the same time. I think
I have forgotten this before.
^ permalink raw reply
* Re: Current Issues #3
From: Daniel Barkalow @ 2006-05-22 21:54 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7v8xoue9eo.fsf@assigned-by-dhcp.cox.net>
On Mon, 22 May 2006, Junio C Hamano wrote:
> * reflog
>
> I still haven't merged this series to "next" -- I do not have
> much against what the code does, but I am unconvinced if it is
> useful. Also objections raised on the list that this can be
> replaced by making sure that a repository that has hundreds of
> tags usable certainly have a point.
I think it would make gitweb's summary view clearer, and Linus seemed
interested in being able to look up what happened in the fast forward
which was the first of several merges in a day.
It could be replaced by a repository with hundreds of machine-readable
tags with code to parse dates into queries for suitable tags. But I don't
think there's an advantage to using the tag mechanism here, because you
never want to look the history up by exactly which history it is (the
thing that a tag ref is good for); you'll be looking for whatever reflog
item is the newest not after a specified time, where the specified time is
almost never a time that a reflog item was created.
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: irc usage..
From: Donnie Berkholz @ 2006-05-22 21:48 UTC (permalink / raw)
To: Linus Torvalds
Cc: Martin Langhoff, Yann Dirson, Git Mailing List, Matthias Urlichs,
Johannes Schindelin
In-Reply-To: <Pine.LNX.4.64.0605221312380.3697@g5.osdl.org>
[-- Attachment #1: Type: text/plain, Size: 233 bytes --]
Linus Torvalds wrote:
> The latest stable CVS release is 1.11.21, I think: you seem to be running
> the "development" version (1.12.x).
Backed down to the 1.11 series, things seem to be going fine so far.
Thanks,
Donnie
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]
^ permalink raw reply
* Re: Local clone/fetch with cogito is glacial
From: H. Peter Anvin @ 2006-05-22 21:18 UTC (permalink / raw)
To: Sean; +Cc: git
In-Reply-To: <BAYC1-PASMTP11FDE05B530CFF43C043E5AE9A0@CEZ.ICE>
Sean wrote:
> On Sun, 21 May 2006 16:47:45 -0700
> "H. Peter Anvin" <hpa@zytor.com> wrote:
>
>> It appears that doing a *local* -- meaning using a file path or file URL
>> -- clone or fetch with cogito is just glacial when the repository has an
>> even moderate number of tags (and it's fetching the tags that takes all
>> the time.) That's a really serious problem for me.
>>
>
> Peter, does git clone work acceptably for you?
>
Well, it does, except it doesn't set up the cogito branches (which one can of course copy
manually.)
cg-clone probably should be rewritten as a thin wrapper around git-clone.
-hpa
^ permalink raw reply
* [PATCH] git status: ignore empty directories (because they cannot be added)
From: Matthias Lederhofer @ 2006-05-22 21:02 UTC (permalink / raw)
To: git
and a new option -u / --untracked-files to show files in untracked
directories.
---
A few things I'm not sure about:
- Should there be another option to disable --no-empty-directory?
- Is the option name --untracked-files ok?
- Should it be documented (probably yes :))? At the moment the
git-status man page does not tell about any command line option at
all but for git-commit it does not make sense.
git-commit.sh | 17 ++++++++++++++---
1 files changed, 14 insertions(+), 3 deletions(-)
---
1921592d5e7809f72a902cca1a38217b150800a9
diff --git a/git-commit.sh b/git-commit.sh
index 6ef1a9d..6785826 100755
--- a/git-commit.sh
+++ b/git-commit.sh
@@ -3,7 +3,7 @@ #
# Copyright (c) 2005 Linus Torvalds
# Copyright (c) 2006 Junio C Hamano
-USAGE='[-a] [-s] [-v] [--no-verify] [-m <message> | -F <logfile> | (-C|-c) <commit>) [--amend] [-e] [--author <author>] [[-i | -o] <path>...]'
+USAGE='[-a] [-s] [-v] [--no-verify] [-m <message> | -F <logfile> | (-C|-c) <commit>] [-u] [--amend] [-e] [--author <author>] [[-i | -o] <path>...]'
SUBDIRECTORY_OK=Yes
. git-sh-setup
@@ -134,13 +134,17 @@ #'
report "Changed but not updated" \
"use git-update-index to mark for commit"
+ option=""
+ if test -z "$untracked_files"; then
+ option="--directory --no-empty-directory"
+ fi
if test -f "$GIT_DIR/info/exclude"
then
- git-ls-files -z --others --directory \
+ git-ls-files -z --others $option \
--exclude-from="$GIT_DIR/info/exclude" \
--exclude-per-directory=.gitignore
else
- git-ls-files -z --others --directory \
+ git-ls-files -z --others $option \
--exclude-per-directory=.gitignore
fi |
perl -e '$/ = "\0";
@@ -203,6 +207,7 @@ verbose=
signoff=
force_author=
only_include_assumed=
+untracked_files=
while case "$#" in 0) break;; esac
do
case "$1" in
@@ -340,6 +345,12 @@ do
verbose=t
shift
;;
+ -u|--u|--un|--unt|--untr|--untra|--untrac|--untrack|--untracke|--untracked|\
+ --untracked-|--untracked-f|--untracked-fi|--untracked-fil|--untracked-file|\
+ --untracked-files)
+ untracked_files=t
+ shift
+ ;;
--)
shift
break
--
1.3.2
^ permalink raw reply related
* Re: irc usage..
From: Linus Torvalds @ 2006-05-22 20:33 UTC (permalink / raw)
To: Martin Langhoff
Cc: Donnie Berkholz, Yann Dirson, Git Mailing List, Matthias Urlichs,
Johannes Schindelin
In-Reply-To: <Pine.LNX.4.64.0605221256090.3697@g5.osdl.org>
On Mon, 22 May 2006, Linus Torvalds wrote:
>
> Of course, the CVS part is probably a big overhead, so it might not help
> much (I would not be surprised at all if a number of the fork/exec/exit
> things are due to the CVS server starting RCS or something, not due to
> git-cvsimport itself)
Ahh. stracing the CVS server seems to imply that it forks off a subprocess
for every command. It doesn't actually execute any external program, but
just does a fork + muck around in the ,v files + exit.
Maybe one of the changes in the 1.12.x versions is to not do that, which
might explain why Donnie seems to see much better performance, but also
sees all the memory leakage?
Linus
^ permalink raw reply
* Re: irc usage..
From: Linus Torvalds @ 2006-05-22 20:20 UTC (permalink / raw)
To: Donnie Berkholz
Cc: Martin Langhoff, Yann Dirson, Git Mailing List, Matthias Urlichs,
Johannes Schindelin
In-Reply-To: <447215D4.5020403@gentoo.org>
On Mon, 22 May 2006, Donnie Berkholz wrote:
>
> Linus Torvalds wrote:
> > Hmm. My cvs server doesn't really grow at all. It's at 13M RSS.
>
> Yeah, that's the thing. RSS stayed about the same (according to top),
> but virtual just kept growing.
Not for me. The virtual size is certainly bigger than RSS, but not by a
huge amount. So this might be a regression in CVS, since you seem to have
a newer version than I do.
The latest stable CVS release is 1.11.21, I think: you seem to be running
the "development" version (1.12.x).
Linus
^ permalink raw reply
* Re: irc usage..
From: Donnie Berkholz @ 2006-05-22 20:16 UTC (permalink / raw)
To: Donnie Berkholz
Cc: Martin Langhoff, Linus Torvalds, Yann Dirson, Git Mailing List,
Matthias Urlichs, Johannes Schindelin
In-Reply-To: <44720C66.6040304@gentoo.org>
[-- Attachment #1: Type: text/plain, Size: 652 bytes --]
Donnie Berkholz wrote:
> OK, I started a new run without -L, and I'm watching it in top right
> now.
Tried a run with -L 1024 and it broke in just a couple of minutes:
Fetching
sys-kernel/linux/files/2.4.0.8/linux-2.4.0-ac8-reiserfs-3.6.25-nfs.diff.gz
v 1.1
New
sys-kernel/linux/files/2.4.0.8/linux-2.4.0-ac8-reiserfs-3.6.25-nfs.diff.gz:
6367 bytes
Tree ID 457f629df10e70a5ef430f431eca27ed02a83d46
Parent ID 0541d8b54a02df3be50d529497236556c6862a4c
Committed patch 1024 (origin 2001-01-13 00:29:39)
Commit ID ba9d995d12a37502a851e198b67e141623f79544
DONE; creating master branch
cat: write error: Broken pipe
Thanks,
Donnie
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]
^ permalink raw reply
* Re: irc usage..
From: Linus Torvalds @ 2006-05-22 20:11 UTC (permalink / raw)
To: Martin Langhoff
Cc: Donnie Berkholz, Yann Dirson, Git Mailing List, Matthias Urlichs,
Johannes Schindelin
In-Reply-To: <46a038f90605221241x58ffa2a4o26159d38d86a8092@mail.gmail.com>
On Tue, 23 May 2006, Martin Langhoff wrote:
>
> The dev machine where I am running the import is a slug! It's still
> working on it, only gotten to 7700 commits, with the cvsimport process
> stable at 28MB RAM and cvs stable at 4MB.
I have to say, that cvsimport script really does do horrible things. It's
basically a fork/exec/exit benchmark, as far as I can tell. Running
oprofile on the thing, the top offenders are (ignore the 45% idle thing:
it's just because this was run on a dual-cpu system, so since it's almost
completely single-threaded you get ~50% idle by default).
3117654 45.8708 vmlinux vmlinux .power4_idle
802313 11.8046 vmlinux vmlinux .unmap_vmas
632913 9.3122 vmlinux vmlinux .copy_page_range
150359 2.2123 vmlinux vmlinux .release_pages
131330 1.9323 vmlinux vmlinux .vm_normal_page
117836 1.7337 libperl.so libperl.so (no symbols)
74098 1.0902 libgklayout.so libgklayout.so (no symbols)
54680 0.8045 vmlinux vmlinux .free_pages_and_swap_cache
54300 0.7989 libfb.so libfb.so (no symbols)
49052 0.7217 vmlinux vmlinux .copy_4K_page
46559 0.6850 libc-2.4.so libc-2.4.so getc
42677 0.6279 vmlinux vmlinux .page_remove_rmap
41133 0.6052 libc-2.4.so libc-2.4.so ferror
..
those kernel functions are all about process create/exit, and COW faulting
after the fork.
Now, this is on ppc, so process creation is likely slower (idiotic PPC VM
page table hashes), but Linux is actually very good at doing this, and the
fact that process create/exit is so high is a very big sign that the
script just ends up executing a _ton_ of small simple processes that do
almost nothing.
I wonder why those "git-update-index" calls seem to be (assuming I read
the perl correctly) done only a few files at a time. We can do a hundreds
in one go, but it seems to want to do just ten files or something at the
same time. Although since most commits should hopefully just modify a
couple of files, that probably isn't a big deal.
That thing would probably be an order of magnitude faster if written to
use the git library interfaces directly. Of course, the CVS part is
probably a big overhead, so it might not help much (I would not be
surprised at all if a number of the fork/exec/exit things are due to the
CVS server starting RCS or something, not due to git-cvsimport itself)
Linus
^ permalink raw reply
* Re: irc usage..
From: Donnie Berkholz @ 2006-05-22 19:49 UTC (permalink / raw)
To: Linus Torvalds
Cc: Martin Langhoff, Yann Dirson, Git Mailing List, Matthias Urlichs,
Johannes Schindelin
In-Reply-To: <Pine.LNX.4.64.0605221234430.3697@g5.osdl.org>
[-- Attachment #1: Type: text/plain, Size: 586 bytes --]
Linus Torvalds wrote:
> Hmm. My cvs server doesn't really grow at all. It's at 13M RSS.
Yeah, that's the thing. RSS stayed about the same (according to top),
but virtual just kept growing.
> What version of cvs are you running?
>
> [torvalds@g5 ~]$ cvs --version
>
> Concurrent Versions System (CVS) 1.11.21 (client/server)
Concurrent Versions System (CVS) 1.12.12 (client/server)
Looks like there's a .13 out but the zlib interaction is badly broken
(-z >=1) so my system didn't get upgraded. I'll try it anyway after the
-L run finishes.
Thanks,
Donnie
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]
^ permalink raw reply
* Re: irc usage..
From: Martin Langhoff @ 2006-05-22 19:46 UTC (permalink / raw)
To: Linus Torvalds
Cc: Donnie Berkholz, Yann Dirson, Git Mailing List, Matthias Urlichs,
Johannes Schindelin
In-Reply-To: <Pine.LNX.4.64.0605221013020.3697@g5.osdl.org>
On 5/23/06, Linus Torvalds <torvalds@osdl.org> wrote:
> Ok, initial results are promising. git-cvsimport appears to be still
> slowly growing, but it's at 40M (ie pretty tiny, considering that cvsps
> grew to 800+MB on this archive) and growth seems to actually be slowing.
That's great news. The cvs archive seems to have large commits every
once in a while, so I suspect the residual memory growth may be
related to those. Or to a smaller leak I haven't nailed.
My test box is bloody slow it seems. I'll try and get hold of a faster
machine to run this if I can.
> As to packing, it doing something like
Given that we are running batch, it is safe and simple to stop the
import, repack, prune-packed, and keep going. Don't think we'll win
any races by running it in parallel ;-)
cheers,
martin
^ permalink raw reply
* Re: irc usage..
From: Martin Langhoff @ 2006-05-22 19:41 UTC (permalink / raw)
To: Donnie Berkholz
Cc: Linus Torvalds, Yann Dirson, Git Mailing List, Matthias Urlichs,
Johannes Schindelin
In-Reply-To: <44720C66.6040304@gentoo.org>
On 5/23/06, Donnie Berkholz <spyderous@gentoo.org> wrote:
> So it seems the problem is in cvs itself. I will try another run with -L
> now.
What version of cvs are you using? Perhaps trying a different one?
The dev machine where I am running the import is a slug! It's still
working on it, only gotten to 7700 commits, with the cvsimport process
stable at 28MB RAM and cvs stable at 4MB.
cheers,
martin
^ permalink raw reply
* Re: irc usage..
From: Linus Torvalds @ 2006-05-22 19:38 UTC (permalink / raw)
To: Donnie Berkholz
Cc: Martin Langhoff, Yann Dirson, Git Mailing List, Matthias Urlichs,
Johannes Schindelin
In-Reply-To: <44720C66.6040304@gentoo.org>
On Mon, 22 May 2006, Donnie Berkholz wrote:
>
> OK, I started a new run without -L, and I'm watching it in top right
> now. The cvsimport seems to be doing alright, but the cvs server process
> sucks about another megabyte of virtual every 4-5 seconds. This is a bit
> concerning since I don't have any swap. Shortly after it hit 670M, I got
> "Cannot allocate memory" again. I've got a gig of RAM, and around 300M
> was resident in various processes at the time.
Hmm. My cvs server doesn't really grow at all. It's at 13M RSS.
What version of cvs are you running?
[torvalds@g5 ~]$ cvs --version
Concurrent Versions System (CVS) 1.11.21 (client/server)
maybe that matters.
(but my import is only up to Jun 22, 2003 so far).
Linus
^ permalink raw reply
* [PATCH] Problem: 'trap...exit' causes error message when /bin/sh is ash.
From: Yakov Lerner @ 2006-05-22 19:34 UTC (permalink / raw)
To: git; +Cc: iler.ml
Problem: 'trap...exit' causes error message when /bin/sh is ash.
Fix: Change 'trap...exit' to 'trap...0' like in other scripts.
Signed-off-by: Yakov Lerner <iler.ml@gmail.com>
---
git-clone.sh | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
954e49bc242cacd27e002f194d54a6895e64f88c
diff --git a/git-clone.sh b/git-clone.sh
index 227245c..d96894d 100755
--- a/git-clone.sh
+++ b/git-clone.sh
@@ -199,7 +199,7 @@ # Try using "humanish" part of source re
[ -e "$dir" ] && echo "$dir already exists." && usage
mkdir -p "$dir" &&
D=$(cd "$dir" && pwd) &&
-trap 'err=$?; cd ..; rm -r "$D"; exit $err' exit
+trap 'err=$?; cd ..; rm -r "$D"; exit $err' 0
case "$bare" in
yes) GIT_DIR="$D" ;;
*) GIT_DIR="$D/.git" ;;
@@ -407,5 +407,5 @@ Pull: refs/heads/$head_points_at:$origin
fi
rm -f "$GIT_DIR/CLONE_HEAD" "$GIT_DIR/REMOTE_HEAD"
-trap - exit
+trap - 0
--
1.3.GIT
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox