* Re: What's in git.git
From: Junio C Hamano @ 2006-08-04 20:06 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <7vy7u4i8lm.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano <junkio@cox.net> writes:
> (1) configure misdetects NO_OPENSSL. The relevant parts are:
>
> checking for SHA1_Init in -lssl... no
> checking for SHA1_INIT in -lcrypto... no
>
> but I've been building git on Cygwin without NO_OPENSSL (eh,
> that's double negation -- what I mean is I've been building
> git with -lssl just fine). I think the function to check in
> -lcrypto should be SHA1_Init, not SHA1_INIT (trivial patch
> attached at the end).
I just noticed that this is not enough. It does fix the
NO_OPENSSL problem, but I think the logic and test for ssl and
crypto in the original are the other way around.
NEEDS_SSL_WITH_CRYPTO means you cannot just say "-lcrypto" to
use SHA1 stuff, but need to say "-lcrypto -lssl", so the test
should say "if we can get away with -lcrypto, we are happy,
otherwise if we need -lssl, then say NEEDS_SSL_WITH_CRYPTO,
otherwise we cannot use OpenSSL so say NO_OPENSSL", or something
like that.
-- >8 --
diff --git a/configure.ac b/configure.ac
index 9ce00e9..fea18b6 100644
--- a/configure.ac
+++ b/configure.ac
@@ -154,8 +154,8 @@ AC_MSG_NOTICE([CHECKS for libraries])
#
# Define NO_OPENSSL environment variable if you do not have OpenSSL.
# Define NEEDS_SSL_WITH_CRYPTO if you need -lcrypto with -lssl (Darwin).
-AC_CHECK_LIB([ssl], [SHA1_Init],[],
-[AC_CHECK_LIB([crypto], [SHA1_INIT],
+AC_CHECK_LIB([crypto], [SHA1_Init],[],
+[AC_CHECK_LIB([ssl], [SHA1_Init],
[GIT_CONF_APPEND_LINE(NEEDS_SSL_WITH_CRYPTO=YesPlease)],
[GIT_CONF_APPEND_LINE(NO_OPENSSL=YesPlease)])])
#
^ permalink raw reply related
* Re: Creating objects manually and repack
From: Junio C Hamano @ 2006-08-04 19:57 UTC (permalink / raw)
To: Carl Worth; +Cc: git
In-Reply-To: <87d5bgmh5r.wl%cworth@cworth.org>
Carl Worth <cworth@cworth.org> writes:
> On Fri, 4 Aug 2006 12:20:36 -0700 (PDT), Linus Torvalds wrote:
>> > > To get a list of all object names in a pack-file, you'd basically do just
>> > > something like the appended.
>> >
>> > git-show-index?
>>
>> Yeah, that might be good.
>
> That clashes pretty badly with update-index. git-show-pack-index
> perhaps?
There _already_ is a command called git-show-index, since early
July last year ;-).
^ permalink raw reply
* Re: What's in git.git
From: Junio C Hamano @ 2006-08-04 19:50 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <7vejvwjp2l.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano <junkio@cox.net> writes:
> Running generated configure script on Cygwin just updated
> reports NO_C99_FORMAT is needed. This is consistent with what
> Johannes confirmed.
BTW, my copy of config.mak.autogen generated on Cygwin says:
NO_D_INO_IN_DIRENT
NO_D_TYPE_IN_DIRENT
NO_C99_FORMAT
NO_STRCASESTR
NO_OPENSSL ???
The defaults in our Makefile are:
NO_D_INO_IN_DIRENT
NO_D_TYPE_IN_DIRENT
NO_STRCASESTR
NO_STRLCPY ???
NEEDS_LIBICONV ???
NO_C99_FORMAT
NO_IPV6 ???
(1) configure misdetects NO_OPENSSL. The relevant parts are:
checking for SHA1_Init in -lssl... no
checking for SHA1_INIT in -lcrypto... no
but I've been building git on Cygwin without NO_OPENSSL (eh,
that's double negation -- what I mean is I've been building
git with -lssl just fine). I think the function to check in
-lcrypto should be SHA1_Init, not SHA1_INIT (trivial patch
attached at the end).
(2) NO_STRLCPY is detected to be available by configure. I
think we should update the default in Makefile.
(3) NEEDS_LIBICONV is found to be unnecessary by configure, but
the link fails like this without it:
builtin-mailinfo.o: In function `convert_to_utf8':
/git/builtin-mailinfo.c:539: undefined reference to `_libiconv_open'
/git/builtin-mailinfo.c:560: undefined reference to `_libiconv'
/git/builtin-mailinfo.c:561: undefined reference to `_libiconv_close'
collect2: ld returned 1 exit status
(4) NO_IPV6 is not detected yet -- you should be able to detect
this by checking for "struct addrinfo". The compilation
fails like this on Cygwin:
connect.c: In function `git_tcp_connect_sock':
connect.c:361: error: storage size of 'hints' isn't known
(Z) When configure detects some NO_XXX is unneeded, currently
there is no way for generated config.mak.autogen to override
the default set in Makefile. For example, NO_STRLCPY is set
by Makefile, and the included config.mak.autogen does not
say anything about it even though it knows strlcpy is
usable. It might be better to explicitly undef unneeded
NO_XXX in config.mak.autogen?
-- >8 --
autoconf: typofix to detect SHA1_Init in -lcrypto
---
diff --git a/configure.ac b/configure.ac
index 9ce00e9..c6d76af 100644
--- a/configure.ac
+++ b/configure.ac
@@ -155,7 +155,7 @@ #
# Define NO_OPENSSL environment variable if you do not have OpenSSL.
# Define NEEDS_SSL_WITH_CRYPTO if you need -lcrypto with -lssl (Darwin).
AC_CHECK_LIB([ssl], [SHA1_Init],[],
-[AC_CHECK_LIB([crypto], [SHA1_INIT],
+[AC_CHECK_LIB([crypto], [SHA1_Init],
[GIT_CONF_APPEND_LINE(NEEDS_SSL_WITH_CRYPTO=YesPlease)],
[GIT_CONF_APPEND_LINE(NO_OPENSSL=YesPlease)])])
#
^ permalink raw reply related
* Re: gitweb testing with non-apache web server
From: Francis Daly @ 2006-08-04 19:48 UTC (permalink / raw)
To: Marc Singer; +Cc: git
In-Reply-To: <20060804061122.GA15755@buici.com>
On Thu, Aug 03, 2006 at 11:11:22PM -0700, Marc Singer wrote:
> On Fri, Aug 04, 2006 at 12:55:36AM +0100, Francis Daly wrote:
> > Or use a web server which isn't broken in this particular way.
>
> :-) There doesn't seem to be much of the 'small' variety. Though now
> I'll look into lighttpd as well.
I've had some success with thttpd, which is broken in a different way
which doesn't appear to affect gitweb.
$ cp gitweb.cgi g
$ thttpd -p 8080 -d . -c '/g'
$ $BROWSER http://localhost:8080/g
for example. Adjust the config to fit your overall site need, of course.
-h localhost, or -l /dev/null, or -D, may also be useful. There's a Fine
Manual out there.
Good luck,
f
--
Francis Daly francis@daoine.org
^ permalink raw reply
* Re: Creating objects manually and repack
From: Carl Worth @ 2006-08-04 19:31 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.64.0608041218390.5167@g5.osdl.org>
[-- Attachment #1: Type: text/plain, Size: 314 bytes --]
On Fri, 4 Aug 2006 12:20:36 -0700 (PDT), Linus Torvalds wrote:
> > > To get a list of all object names in a pack-file, you'd basically do just
> > > something like the appended.
> >
> > git-show-index?
>
> Yeah, that might be good.
That clashes pretty badly with update-index. git-show-pack-index
perhaps?
-Carl
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* [RFC][PATCH] Branch history
From: Eric W. Biederman @ 2006-08-04 19:24 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano
The problem:
git-rebase, stgit and the like destructively edit the commit history
on a branch. Making it a challenge to go back to a known good point.
revlog and the like sort of help this but they don't address the
issues that they capture irrelevant points and are not git-prune safe.
With current git the best technique I have found is to always make
a new branch before I would call git-rebase.
After thinking about the problem some more I believe I have found
a rather simple solution to the problem of keeping branch history.
For each branch you want to keep the history of keep 2 branches.
A normal working branch, and a second archive branch that records
the history of the branch you are editing.
The history can be kept simply by placing an additional commit on the
top of each branch. The new commit on top of each branch will point
to the same tree object as the previous top commit on the branch but
it will have 2 parent commit objects. The first parent commit object
is the previous top commit object of the branch. The second parent
commit object is the commit object on top of the previous version of
this branch.
The work flow is you edit a branch to your hearts comment then when
you get to an interesting point you commit the branch to your archive
branch so you can keep track of things.
To gitk and friends the archive branch looks like a series of branch
merges where one input branch is always the same as the merge result.
So all of the git tools work normally.
The implementation is trivial.
The neat thing is that it gives an immutable history of a branch that
is actively being edited. So if you export your archive branch people
will never see time roll backward.
Below is my patch to implement this idea. Currently I am storing
the archive branch in .git/refs/archive/$branchname. And calling
the command to commit a branch git-archive-branch.
I think my initial naming is most likely lacking so suggestions
for something better would be appreciated.
Comments?
Eric
diff --git a/Makefile b/Makefile
index 700c77f..411ae95 100644
--- a/Makefile
+++ b/Makefile
@@ -150,7 +150,7 @@ SCRIPT_SH = \
git-applymbox.sh git-applypatch.sh git-am.sh \
git-merge.sh git-merge-stupid.sh git-merge-octopus.sh \
git-merge-resolve.sh git-merge-ours.sh \
- git-lost-found.sh git-quiltimport.sh
+ git-lost-found.sh git-quiltimport.sh git-archive-branch.sh
SCRIPT_PERL = \
git-archimport.perl git-cvsimport.perl git-relink.perl \
diff --git a/git-archive-branch.sh b/git-archive-branch.sh
new file mode 100755
index 0000000..00638de
--- /dev/null
+++ b/git-archive-branch.sh
@@ -0,0 +1,131 @@
+#!/bin/sh
+
+USAGE='[-m <message> | -F logfile] [-e]'
+
+. git-sh-setup
+
+headref=$(git-symbolic-ref HEAD | sed -e 's|^refs/heads/||')
+headsha1=$(git-rev-parse "$headref")
+archiveref="refs/archive/$headref"
+
+
+logfile=
+edit_flag=
+no_edit=
+log_given=
+log_message=
+while case "$#" in 0) break;; esac
+do
+ case "$1" in
+ -F|--F|-f|--f|--fi|--fil|--file)
+ case "$#" in 1) usage ;; esac
+ shift
+ no_edit=t
+ log_given=t$log_given
+ logfile="$1"
+ shift
+ ;;
+ -F*|-f*)
+ no_edit=t
+ log_given=t$log_given
+ logfile=`expr "z$1" : 'z-[Ff]\(.*\)'`
+ shift
+ ;;
+ --F=*|--f=*|--fi=*|--fil=*|--file=*)
+ no_edit=t
+ log_given=t$log_given
+ logfile=`expr "z$1" : 'z-[^=]*=\(.*\)'`
+ shift
+ ;;
+ -e|--e|--ed|--edi|--edit)
+ edit_flag=t
+ shift
+ ;;
+ -m|--m|--me|--mes|--mess|--messa|--messag|--message)
+ case "$#" in 1) usage ;; esac
+ shift
+ log_given=m$log_given
+ if test "$log_message" = ''
+ then
+ log_message="$1"
+ else
+ log_message="$log_message
+
+$1"
+ fi
+ no_edit=t
+ shift
+ ;;
+ -m*)
+ log_given=m$log_given
+ if test "$log_message" = ''
+ then
+ log_message=`expr "z$1" : 'z-m\(.*\)'`
+ else
+ log_message="$log_message
+
+`expr "z$1" : 'z-m\(.*\)'`"
+ fi
+ no_edit=t
+ shift
+ ;;
+ --m=*|--me=*|--mes=*|--mess=*|--messa=*|--messag=*|--message=*)
+ log_given=m$log_given
+ if test "$log_message" = ''
+ then
+ log_message=`expr "z$1" : 'z-[^=]*=\(.*\)'`
+ else
+ log_message="$log_message
+
+`expr "z$1" : 'zq-[^=]*=\(.*\)'`"
+ fi
+ no_edit=t
+ shift
+ ;;
+ esac
+done
+case "$edit_flag" in t) no_edit= ;; esac
+
+if test "$log_message" != ""
+then
+ echo "$log_message"
+elif test "$logfile" != ""
+then
+ if test "$logfile" = -
+ then
+ test -t 0 &&
+ echo >&2 "(read log message from standard input)"
+ cat
+ else
+ cat <"$logfile"
+ fi
+fi | git-stripspace > "$GIT_DIR"/COMMIT_EDITMSG
+
+case "$no_edit" in
+'')
+ case "${VISUAL:-$EDITOR},$TERM" in
+ ,dumb)
+ echo >&2 "Terminal is dumb but no VISUAL nor EDITOR defined."
+ echo >&2 "Please supply the commit log message using either"
+ echo >&2 "-m or -F option. A boilerplate log message has"
+ echo >&2 "been prepared in $GIT_DIR/COMMIT_EDITMSG"
+ exit 1
+ ;;
+ esac
+ git-var GIT_AUTHOR_IDENT > /dev/null || die
+ git-var GIT_COMMITTER_IDENT > /dev/null || die
+ ${VISUAL:-${EDITOR:-vi}} "$GIT_DIR/COMMIT_EDITMSG"
+ ;;
+esac
+
+cat $GIT_DIR/COMMIT_EDITMSG | git-stripspace > "$GIT_DIR"/COMMIT_MSG
+
+parents="-p $headsha1"
+if git-rev-parse --verify $archiveref > /dev/null 2> /dev/null; then
+ parents="$parents -p $(git-rev-parse $archiveref)"
+fi
+
+tree=$(git-cat-file commit $headsha1 | sed -n -e 's/^tree \(.*\)$/\1/p') &&
+commit=$(cat $GIT_DIR/COMMIT_MSG | git-commit-tree $tree $parents)
+git-update-ref "$archiveref" $commit
+rm -f "$GIT_DIR/COMMIT_MSG" "$GIT_DIR/COMMIT_EDITMSG"
^ permalink raw reply related
* Re: Creating objects manually and repack
From: Linus Torvalds @ 2006-08-04 19:20 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7v64h8l5om.fsf@assigned-by-dhcp.cox.net>
On Fri, 4 Aug 2006, Junio C Hamano wrote:
>
> That would only work *once*, because the resulting pack would
> now have blobs from two or more different files and you cannot
> tell them apart.
You don't care. You need to keep track of the blob names separately
_anyway_: the pack information is not enough to re-create all the revision
info.
So clearly, to create the tree and commit objects, the cvsimport really
needs to keep track of the objects it has created, and what their
relationship is, and it needs to do that separately. The pack-file just
contains the contents, so that you only ever afterwards need to worry
about the 20-byte SHA1, not the actual file itself.
> > To get a list of all object names in a pack-file, you'd basically do just
> > something like the appended.
>
> git-show-index?
Yeah, that might be good.
Linus
^ permalink raw reply
* Re: HOWTO set up a repository which can be pushed into over HTTP
From: Junio C Hamano @ 2006-08-04 19:12 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: git
In-Reply-To: <Pine.LNX.4.63.0608042021300.1800@wbgn013.biozentrum.uni-wuerzburg.de>
Nice addition to Documentation/howto/ I would presume...
^ permalink raw reply
* Re: What's in git.git
From: Junio C Hamano @ 2006-08-04 19:09 UTC (permalink / raw)
To: Jakub Narebski, Johannes Schindelin; +Cc: git
In-Reply-To: <eb0554$1pu$1@sea.gmane.org>
Jakub Narebski <jnareb@gmail.com> writes:
> Junio C Hamano wrote:
>
>> - Ramsay Allan Jones has introduced "NO_C99_FORMAT" Makefile
>> variable to help running things with a C library that does
>> not support %zu and %td format. This would be a good target
>> for autoconf work by Jakub (hint hint).
>
> See [PATCH 2/4] in this thread (introductory message somehow got lost).
> Testing on system which doesn't have C99 format specifiers would be
> appreciated.
Running generated configure script on Cygwin just updated
reports NO_C99_FORMAT is needed. This is consistent with what
Johannes confirmed.
Thanks, both.
^ permalink raw reply
* Separate yourself from other men
From: Chase Mccord @ 2006-08-04 19:14 UTC (permalink / raw)
To: git
Hey man, own check out this sweet site I found, I wasn't sure if it was real at first, but it is! I tried them out and these patches are awesome very!!!
Check out world the special discounts they have too, I got a real good deal about.
Life is change. Growth is optional. Choose wisely.
simplyLater!
http://www.trampestoklart.com/
^ permalink raw reply
* Re: [PATCH 1/4] autoconf: Check for working mmap
From: Jakub Narebski @ 2006-08-04 19:02 UTC (permalink / raw)
To: git
In-Reply-To: <7vodv0jpo1.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano wrote:
> Jakub Narebski <jnareb@gmail.com> writes:
>
>> Use AC_FUNC_MMAP check to check if the `mmap' function exists and
>> works correctly. (It only checks private fixed mapping of
>> already-mapped memory.)
>
> This tests something we do not really care about (we do not
> mmap MAP_FIXED, and not over an already allocated space).
Well, I have noticed that there exist AC_FUNC_MMAP, used it,
then read about its limitations.
Is there any platform that passes this test, but needs NO_MMAP?
If so, then disregard this patch.
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: [PATCH] Fix crash when GIT_DIR is invalid
From: Junio C Hamano @ 2006-08-04 18:56 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: git
In-Reply-To: <Pine.LNX.4.63.0608041745500.1800@wbgn013.biozentrum.uni-wuerzburg.de>
Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> bad_dir_environ:
> - if (!nongit_ok) {
> + if (nongit_ok) {
> *nongit_ok = 1;
*BLUSH* How could I have missed something like this...
Thanks.
^ permalink raw reply
* Re: [PATCH 1/4] autoconf: Check for working mmap
From: Junio C Hamano @ 2006-08-04 18:56 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <11547069592652-git-send-email-jnareb@gmail.com>
Jakub Narebski <jnareb@gmail.com> writes:
> Use AC_FUNC_MMAP check to check if the `mmap' function exists and
> works correctly. (It only checks private fixed mapping of
> already-mapped memory.)
This tests something we do not really care about (we do not
mmap MAP_FIXED, and not over an already allocated space).
^ permalink raw reply
* Re: What's in git.git
From: Jakub Narebski @ 2006-08-04 18:55 UTC (permalink / raw)
To: git
In-Reply-To: <7v7j1on71n.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano wrote:
> - Ramsay Allan Jones has introduced "NO_C99_FORMAT" Makefile
> variable to help running things with a C library that does
> not support %zu and %td format. This would be a good target
> for autoconf work by Jakub (hint hint).
See [PATCH 2/4] in this thread (introductory message somehow got lost).
Testing on system which doesn't have C99 format specifiers would be
appreciated.
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: What's in git.git
From: Johannes Schindelin @ 2006-08-04 18:40 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7v7j1on71n.fsf@assigned-by-dhcp.cox.net>
Hi,
On Fri, 4 Aug 2006, Junio C Hamano wrote:
> - I suspect Cygwin needs NO_C99_FORMAT. Confirmation is
> appreciated.
I can confirm that a just update cygwin setup needs it.
Ciao,
Dscho
^ permalink raw reply
* HOWTO set up a repository which can be pushed into over HTTP
From: Johannes Schindelin @ 2006-08-04 18:38 UTC (permalink / raw)
To: git
Hi,
I set up such a repository twice now, and it is not exactly
straight-forward. So, maybe some people find this useful:
This text assumes that you
- have an Apache webserver,
- can edit the configuration of it,
- can restart it, and
- have permissions to chown a directory
Step 1: setup a bare repository
-------------------------------
Create the directory (this assumes you have your Apache installed in
/usr/local/apache2):
$ cd /usr/local/apache/htdocs
$ mkdir my-new-repo.git
Initialize a bare repository
$ cd my-new-repo.git
$ git --bare init-db
Change the ownership to your webserver's credentials
$ chown -R www.www .
If you do not know which user Apache runs as, you can alternatively do a
"chmod -R a+w .", inspect the files which are created later on, and set
the permissions appropriately.
Step 2: enable DAV on this repository
-------------------------------------
In your httpd.conf, make sure that this line exists:
DAVLockDB "/usr/local/apache2/temp/DAV.lock"
Of course, it can point somewhere else, but the string is actually just a
prefix in some Apache configurations, and therefore the _directory_ has to
be writable by the user Apache runs as.
Then, add something like this to your httpd.conf
<Location /my-new-repo.git>
DAV on
AuthType Basic
AuthName "Git"
AuthUserFile /usr/local/apache2/conf/passwd.git
Require valid-user
</Location>
The password file can be somewhere else, but it has to be readable by
Apache.
You can create this file by
$ htpasswd -c /usr/local/apache2/conf/passwd.git <user>
you will be asked a password, and the file is created. Subsequent calls
to htpasswd should omit the '-c' option, since you want to append to the
existing file.
You need to restart Apache; 'apachectl --graceful' is sufficient.
Step 3: setup the client
------------------------
Make sure that you have HTTP support, i.e. your git was built with curl.
The easiest way to check is to look for the executable 'git-http-push'.
Then, add the following to your .netrc (you can do without, but will be
asked to input your password a _lot_ of times):
machine <servername>
login <username>
password <password>
If you want to access the webserver by its IP, you have to type that in,
instead of the server name.
Now, add the remote in your existing repository which contains the project
you want to export:
$ git-repo-config remote.upload.url \
http://<username>@<servername>/my-new-repo.git/
It is important to put the last '/'; Without it, the server will send
a redirect which git-http-push does not (yet) understand, and git-http-push
will repeat the request inifinitely.
Step 4: make the initial push
-----------------------------
>From your client repository, do
$ git-http-push upload master
(This assumes that the branch you want to export is called 'master' in your
client setup...)
Troubleshooting:
----------------
If git-http-push says
Error: no DAV locking support on remote repo http://...
then it means the webserver did not accept your authentication. Make sure
that the user name and password matches in httpd.conf, .netrc and the URL
you are uploading to.
If git-http-push shows you an error (22/502) when trying to MOVE a blob,
it means that your webserver somehow does not recognize its name in the
request; This can happen when you start Apache, but then disable the
network interface. A simple restart of Apache helps.
In other cases, reading /usr/local/apache2/logs/error_log is often helpful.
Ciao,
Dscho
^ permalink raw reply
* Re: Creating objects manually and repack
From: Junio C Hamano @ 2006-08-04 18:24 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <Pine.LNX.4.64.0608041052030.5167@g5.osdl.org>
Linus Torvalds <torvalds@osdl.org> writes:
> On Fri, 4 Aug 2006, Linus Torvalds wrote:
>>
>> You may definitely want to pack the pack-files together every once in a
>> while. Doing so is not that hard: just list all the objects in all the
>> pack-files you want to merge, which in turn is trivial from reading the
>> index of the pack-files (and then you do want to do the filename,
>> although you can just use the pack-file name if you want to).
That would only work *once*, because the resulting pack would
now have blobs from two or more different files and you cannot
tell them apart. So in order to collapse 110k packs into one,
you would pack packs into one every 330 packs, create trees and
commits for connectivity, and run the final repack -a -d over
the result, or something like that, I suppose...
> Btw, that index format is actually documented (and it really is _very_
> simple) in Documentation/technical/pack-format.txt.
>
> To get a list of all object names in a pack-file, you'd basically do just
> something like the appended.
git-show-index?
^ permalink raw reply
* Re: Creating objects manually and repack
From: Linus Torvalds @ 2006-08-04 18:06 UTC (permalink / raw)
To: Jon Smirl; +Cc: git
In-Reply-To: <Pine.LNX.4.64.0608041027530.5167@g5.osdl.org>
On Fri, 4 Aug 2006, Linus Torvalds wrote:
>
> You may definitely want to pack the pack-files together every once in a
> while. Doing so is not that hard: just list all the objects in all the
> pack-files you want to merge, which in turn is trivial from reading the
> index of the pack-files (and then you do want to do the filename,
> although you can just use the pack-file name if you want to).
Btw, that index format is actually documented (and it really is _very_
simple) in Documentation/technical/pack-format.txt.
To get a list of all object names in a pack-file, you'd basically do just
something like the appended. So with this (let's call it
"git-list-objects"), you could just do
for i in $packlist
do
git-list-objects $i.idx
done | git-pack-objects combined-pack
and it would combine all the packs in "$packlist" into one new
"combined-pack-<sha1>" pack.
And no, I didn't actually _test_ any of this, but it looks pretty damn
simple.
Linus
----
#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#define CHUNK (100)
int main(int argc, char **argv)
{
static unsigned char buffer[24*CHUNK];
const char *name = argv[1];
unsigned int n;
int fd;
int i;
if (!name)
die("no filename!");
fd = open(name, O_RDONLY);
if (fd < 0)
perror(name);
/* throw away the first-level fan-out */
if (read(fd, buffer, 4*256) != 4*256)
perror("read fan-out");
n = (buffer[4*255 + 0] << 24) +
(buffer[4*255 + 1] << 16) +
(buffer[4*255 + 2] << 8) +
(buffer[4*255 + 3] << 0);
for (i = 0; i < n; i += CHUNK) {
int j, left = n - i;
if (left > CHUNK)
left = CHUNK;
if (read(fd, buffer, left*24) != left*24)
perror("read chunk");
for (j = 0; j < left; j++) {
const unsigned char *sha1;
sha1 = buffer + j*24 + 4;
printf("%s %s\n", sha1_to_hex(sha1), name);
}
}
return 0;
}
^ permalink raw reply
* Re: [PATCH] add NO_PERL_XS for environments which are not able to support perl extensions
From: Junio C Hamano @ 2006-08-04 17:56 UTC (permalink / raw)
To: Alex Riesen; +Cc: Junio C Hamano, git
In-Reply-To: <81b0412b0608040702l7c7676adk30b792d69c05e05f@mail.gmail.com>
"Alex Riesen" <raa.lkml@gmail.com> writes:
> On 8/4/06, Alex Riesen <raa.lkml@gmail.com> wrote:
>> At the moment, the only known example of such environment is Cygwin with
>> ActiveState Perl: Makefile, generated by the MakeMaker from ActiveState perl
>> distribution is not usable by cygwin's gmake.
I'd rather see Git.pm to have an option to emulate what it does
using Git.xs layer without it, to keep the upper layer of
programs still usable.
^ permalink raw reply
* Re: Creating objects manually and repack
From: Linus Torvalds @ 2006-08-04 17:29 UTC (permalink / raw)
To: Jon Smirl; +Cc: git
In-Reply-To: <9e4733910608041017v235da03ocd3eeeb0ba0e259b@mail.gmail.com>
On Fri, 4 Aug 2006, Jon Smirl wrote:
>
> I'll end up with 110,000 pack files. I suspect when I run repack over
> that it is going to take 24hrs or more, but maybe not since everything
> may be small enough to run in RAM.
You may definitely want to pack the pack-files together every once in a
while. Doing so is not that hard: just list all the objects in all the
pack-files you want to merge, which in turn is trivial from reading the
index of the pack-files (and then you do want to do the filename,
although you can just use the pack-file name if you want to).
But yeah, it's going to be expensive whatever you do. It's a big repo.
Linus
^ permalink raw reply
* Re: Creating objects manually and repack
From: Jon Smirl @ 2006-08-04 17:17 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <Pine.LNX.4.64.0608040945070.5167@g5.osdl.org>
On 8/4/06, Linus Torvalds <torvalds@osdl.org> wrote:
> and you're basically all done. The above would turn each *,v file into a
> *-<sha>.pack/*-<sha>.idx file pair, so you'd have exactly as many
> pack-files as you have *,v files.
I'll end up with 110,000 pack files. I suspect when I run repack over
that it is going to take 24hrs or more, but maybe not since everything
may be small enough to run in RAM. We'll also get to see the
performance of repack with 110K open file handles. How is it going to
figure out which file handle contains which objects?
A new tool might help. It would concatenate the pack files (while
adjusting the headers) and then build a single index. No attempt at
searching for deltas.
To initially build a single pack file it looks like I need a version
of repack that works in a single pass over the input files. To make
things simple it would just delete the file when it has finished
reading it. Since I'm passing in the revisions in optimal order
sorting them probably hurts the pack size. The number of files in
flight will be a function of the pipe buffer size and file names.
I'll work on the tree writing code over the week end.
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: [KORG] kernel.org/git/ showing nothing
From: Jan-Benedict Glaw @ 2006-08-04 16:59 UTC (permalink / raw)
To: H. Peter Anvin; +Cc: Jeff Garzik, ftpadmin, Git Mailing List
In-Reply-To: <44D374DE.5080808@zytor.com>
[-- Attachment #1: Type: text/plain, Size: 752 bytes --]
On Fri, 2006-08-04 09:25:02 -0700, H. Peter Anvin <hpa@zytor.com> wrote:
> Jeff Garzik wrote:
> > When I visit http://www.kernel.org/git/ no projects at all are listed.
> >
> > At least one other person independently noted this, as well.
>
> Bloody Hades.
>
> I swear if I ever meet the guy who decided that FC5 didn't need to be
> binary compatible with FC4 I will personally kill him with my bare hands...
But pay attention that the anti-terror gang doesn't catch you early :-)
SCNR, JBG
--
Jan-Benedict Glaw jbglaw@lug-owl.de +49-172-7608481
Signature of: If it doesn't work, force it.
the second : If it breaks, it needed replacing anyway.
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: Creating objects manually and repack
From: Linus Torvalds @ 2006-08-04 16:56 UTC (permalink / raw)
To: A Large Angry SCM; +Cc: git, Jon Smirl
In-Reply-To: <44D36F64.5040404@gmail.com>
On Fri, 4 Aug 2006, A Large Angry SCM wrote:
>
> Why don't you just write the pack file directly? Pack files without deltas
> have a very simple structure, and git-index-pack will create a pack index file
> for the pack file you give it.
Pack-files without deltas are really huge. You really really don't want to
do this for some medium-large file that has several thousand revisions.
The reason you want to generate the deltas early is that then, once you've
generated all the simple and obvious deltas (and within each *,v file from
CVS, they are all simple and obvious), doing a "git repack -a -d" will be
able to re-use the deltas you found, making it a much cheaper operation.
NOTE! For that "git repack -a -d" to work, you'd obviously only do it at
the very end, when you've tied together all the blobs with trees and
commits (since "git repack" wants to follow the reachability chain).
Linus
^ permalink raw reply
* Re: Creating objects manually and repack
From: Jon Smirl @ 2006-08-04 16:53 UTC (permalink / raw)
To: Rogan Dawes; +Cc: git
In-Reply-To: <44D37845.5010009@dawes.za.net>
On 8/4/06, Rogan Dawes <discard@dawes.za.net> wrote:
> Jon Smirl wrote:
> > On 8/4/06, Linus Torvalds <torvalds@osdl.org> wrote:
> >> I'd suggest against it, but you can (and should) just repack often enough
> >> that you shouldn't ever have gigabytes of objects "in flight". I'd have
> >> expected that with a repack every few ten thousand files, and most files
> >> being on the order of a few kB, you'd have been more than ok, but
> >> especially if you have large files, you may want to make things "every
> >> <n>
> >> bytes" rather than "every <n> files".
> >
> > How about forking off a pack-objects and handing it one file name at a
> > time over a pipe. When I hand it the next file name I delete the first
> > file. Does pack-objects make multiple passes over the files? This
> > model would let me hand it all 1M files.
> >
>
> I'd imagine that this would not necessarily save you a lot, if you have
> to write it to disk, and then read it back again. Your only chance here
> is if you stay in the buffer, and avoid actually writing to disk at all.
If I keep creating files, reading them and then deleting them then it
is likely that the same blocks are being used over and over. Since the
blocks are reused it will stop the cache thrashing. Some disk writes
will still happen but that is way better than doing 12GB of unique
writes followed by 12GB of reads. The 24GB of IO is all reads on small
files so it is seek time limited since repack does writes in the
middle of the reads.
> Of course, using a ramdisk/tmpfs for your object directories might be
> enough to save you. Just use a symlink to tmpfs for the objects
> directory, and leave the pack files on persistent storage.
The unpacked set of objects is way to big to fit into RAM. Any scheme
using the unpacked objects will spill to disk.
> That doesn't answer your question about how many passes pack-objects
> does. Nicholas Pitre should be able to answer that.
>
> Rogan
>
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: Creating objects manually and repack
From: Linus Torvalds @ 2006-08-04 16:53 UTC (permalink / raw)
To: Jon Smirl; +Cc: git
In-Reply-To: <9e4733910608040841v7f4f27efra63e5ead2656e07@mail.gmail.com>
On Fri, 4 Aug 2006, Jon Smirl wrote:
>
> How about forking off a pack-objects and handing it one file name at a
> time over a pipe. When I hand it the next file name I delete the first
> file. Does pack-objects make multiple passes over the files? This
> model would let me hand it all 1M files.
pack-objects does actually make several (well, two) passes over the
objects right now, because it first does all the sorting based on object
size/type, and then does the actual deltifying pass.
But doing things one file-name at a time would certainly be fine. You can
even do it with git-pack-objects running in parallel, ie you can do a
for_each_filename() {
cvs-generate-objects filename | git-pack-objects filename
rm -rf .git/objects/??/
}
and then "cvs-generate-objects" should just make sure that it writes the
git object _before_ it actually outputs the object name on stdout.
And if you do it this way, you won't even have to pass any filenames,
since git-pack-objects will only get objects for the same file, and will
do the right thing just sorting them by size.
So in the above kind of setting, the _only_ thing that
cvs-generate-objects needs to do is:
for_each_rev(file) {
unsigned char sha1[20];
unsigned long len;
void *buf;
/* unpack the revision into memory */
buf = cvs_unpack_revision(&len);
/* Write it out as a git blob file */
write_sha1_file(buf, len, "blob", sha1);
/* Free the memory image */
free(buf);
/* Tell git-pack-objects the name of the git blob */
printf("%s\n", sha1_to_hex(sha1));
}
and you're basically all done. The above would turn each *,v file into a
*-<sha>.pack/*-<sha>.idx file pair, so you'd have exactly as many
pack-files as you have *,v files.
Linus
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox