* Re: git web view doesn't show short log in chronological order ?
From: Andy Whitcroft @ 2006-09-11 9:19 UTC (permalink / raw)
To: Toralf Förster; +Cc: git
In-Reply-To: <200609111101.00801.toralf.foerster@gmx.de>
Toralf Förster wrote:
> http://www.kernel.org/git/?p=linux%2Fkernel%2Fgit%2Ftorvalds%2Flinux-2.6.git;a=summary
> lists the entry from Kirill Korotaev at position 12 (all first 12 positions are
> 2 days old), but Kirill Korotaev did the latest commit (if I do a "$>git log" at
> my local clone of Linus' tree) and therefore should be listed at position 1,
> isn't it ?
The dates on commits can always be suspect as they are generally in the
originators local clock. You also have to remember the date which is
shown is the authors original create time. Not the time it was commited
to linus' tree.
/me pulls the dates for 11 & 12:
Author: Thomas Gleixner <tglx@linutronix.de> 2006-09-08 17:47:15
Committer: Linus Torvalds <torvalds@g5.osdl.org> 2006-09-08 18:22:50
Author: Kirill Korotaev <dev@openvz.org> 2006-09-07 11:17:04
Committer: Linus Torvalds <torvalds@g5.osdl.org> 2006-09-08 16:40:46
From what I can see both dates here are older in Kirill's change. So by
any sort they are in the right order (where right is reverse chronological).
?
-apw
^ permalink raw reply
* Re: Change set based shallow clone
From: Jakub Narebski @ 2006-09-11 9:04 UTC (permalink / raw)
To: git
In-Reply-To: <Pine.LNX.4.64.0609101949580.27779@g5.osdl.org>
Linus Torvalds wrote:
> On Mon, 11 Sep 2006, Paul Mackerras wrote:
>>
>> Do you think there is any way to speed up the cold-cache case for
>> git-ls-remote and git-rev-parse with thousands of heads and tags?
>
> Nothing obvious comes to mind.
>
> If we did the same pack-file approach that we do for objects, the problem
> ends up being that _updating_ things is really hard. What we could do (and
> might work) is that a "git repack" would create a "packed representation
> of the heads too".
Because usually there is reasonable number of heads, and that is the tags
that might be problem, because the number of tags likes to grow with the
history, perhaps we could pack only tags heads into the pack? This has
additional advantage that tags, especially annotated tags (i.e. true tags
objects) changes rarely.
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* git web view doesn't show short log in chronological order ?
From: Toralf Förster @ 2006-09-11 9:00 UTC (permalink / raw)
To: git
[-- Attachment #1: Type: text/plain, Size: 379 bytes --]
http://www.kernel.org/git/?p=linux%2Fkernel%2Fgit%2Ftorvalds%2Flinux-2.6.git;a=summary
lists the entry from Kirill Korotaev at position 12 (all first 12 positions are
2 days old), but Kirill Korotaev did the latest commit (if I do a "$>git log" at
my local clone of Linus' tree) and therefore should be listed at position 1,
isn't it ?
--
MfG/Sincerely
Toralf Förster
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: Dropping Git.pm (at least Git.xs)?
From: Jakub Narebski @ 2006-09-11 8:58 UTC (permalink / raw)
To: git
In-Reply-To: <20060911032557.GF23891@pasky.or.cz>
Petr Baudis wrote:
> Dear diary, on Mon, Sep 11, 2006 at 12:13:01AM CEST, I got a letter
> where Jakub Narebski <jnareb@gmail.com> said that...
>> Could you please put appropriate information on GitWiki
>> http://git.or.cz/gitwiki/InterfacesFrontendsAndTools
>> Perhaps it would be good time to start new section, Git Implementations,
>> and put egit (Java GIT library and Eclipse plugin) there too.
>
> This really isn't a Git reimplementation (thankfully).
Perhaps "language binding" would be better name...
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: Change set based shallow clone
From: Paul Mackerras @ 2006-09-11 8:36 UTC (permalink / raw)
To: Linus Torvalds; +Cc: linux, jonsmirl, git
In-Reply-To: <Pine.LNX.4.64.0609101949580.27779@g5.osdl.org>
Linus Torvalds writes:
> So we could generate a "pack of references", but then any modifications
> would be done with the current loose "file objects" approach, and just
> have the filesystem override the pack-files. The problem then actually
> becomes one of _deleting_ branches, because then we'd have to add a
> "negative branch" loose object. Ugly.
Could we do a cache of the refs that stores the stat information for
each of the files under .git/refs plus the sha1 that the ref points
to? In other words this cache would do for the refs what the index
does for the working directory. Reading all the refs would mean we
still had to stat each of the files, but that's much quicker than
reading them in the cold-cache case. In the common case when most of
the stat information matches, we don't have to read the file because
we have the sha1 that the file contains right there in the cache.
Ideally we would have two sha1 values in the cache - the sha1 in the
file, and if that is the ID of a tag object, we would also put the
sha1 of the commit that the tag points to in the cache.
Paul.
^ permalink raw reply
* Re: Change set based shallow clone
From: Junio C Hamano @ 2006-09-11 7:11 UTC (permalink / raw)
To: Shawn Pearce; +Cc: git
In-Reply-To: <20060911060820.GB29368@spearce.org>
Shawn Pearce <spearce@spearce.org> writes:
> I think Linus was suggesting the same thing, just not nearly as
> obvious. He also was looking for ways to not update it frequently.
> But I can't say I'm a fan of negative entries. :-)
I like Linus's suggestion quite a lot actually. It would be a
nice little project that needs to touch reasonably well-isolated
area of the code, but I suspect it would need a bit more
exposure and familiarity to the existing code than somebody who
looks at the core code for the first time.
> But at this point I have more on my plate than I have time for. :-)
You've done quite a lot of work on refs area yourself, so you
might want to give somebody else the chance to try first ;-).
> I've got to get that map window code merged against master or next
> (preference Junio? next touches code I'm touching too but those
> aren't in master yet) for 32 bit offset.
As you know I've shelved the 64-bit offset stuff. My preference
is to base it on 106d710b but basing it on 'next' would be fine.
Between 106d710b and next there is only one unrelated change to
sha1_file.c, which is the hexval[] change that is in 'master'.
I've sent out "What's in" tonight but forgot to talk about plans
for topics in "next", so here is an update.
- These things are probably Ok, and after a bit more testing
I'd like to push them out to 'master' by my Wednesday:
- Andy Whitcroft's "send-pack to use git-rev-list --stdin"
- "git apply" to apply binary without explicit --binary flag;
- "git diff --binary" to do full-index only for binary patch;
- "git unpack-objects -r", except it needs a better name
(perhaps --recover).
- And these by the end of week:
- Jeff King's git-runstatus;
- Frank and Rene's git-archive along with git-daemon change;
- I am not _so_ comfortable with these quite yet, but plan to
test it a bit more to gain confidence and decide when to push
out to 'master' perhaps by the middle of the week:
- pack-objects --unpacked=this-pack --unpacked=that-pack
- pack-objects --stdin
> I also want to spend more time on the dictionary packing prototype to
> see if its worth spending even more time on.
So far I liked the per-object-type dictionary conjecture the
most, primarily because my gut feeling tells me that it would
involve the least amount of hassle in the interoperability area.
But honestly speaking I am not looking forward to a packfile
format update at this moment. I'd like things to settle down a
bit now, after merging good bits from 'next' to 'master' and
declare 1.4.3 first, perhaps by the end of the month.
^ permalink raw reply
* Re: Change set based shallow clone
From: Junio C Hamano @ 2006-09-11 6:35 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git, Paul Mackerras
In-Reply-To: <Pine.LNX.4.64.0609102002300.27779@g5.osdl.org>
Linus Torvalds <torvalds@osdl.org> writes:
> On Sun, 10 Sep 2006, Linus Torvalds wrote:
>>
>> If we did the same pack-file approach that we do for objects, the problem
>> ends up being that _updating_ things is really hard. What we could do (and
>> might work) is that a "git repack" would create a "packed representation
>> of the heads too".
>
> To clarify: I'm _not_ suggesting actually using the "pack-file"
> representation itself for the references.
>
> I'm saying that we could have something like a
>
> .git/refs-packed
>
> file, which could be (for example) just a plain linear text-file, of the
> form
>
> 19ed368b17abfb9ad5c7467ea74fd8a045d96b43 refs/heads/html
> 60a6bf5f53635005f4f68d8b8a33172309193623 refs/heads/maint
> ...
>
> ie it would contain just a linear file with the "<hex></tab><refname>"
> format. Then, the way to look up a reference would be:
>...
> NOTE! It's important that whatever sceme used gets locking right. The
> above suggestion gets it right simply because it doesn't really _change_
> anything. Any new or modified ref ends up using the old code, and using a
> ".lock" file and renaming it automatically does the same thing it ever
> did.
This is all interesting and can be done by somebody interested
in the really core part of the system.
I was just looking at Paul's "new" branch, and realized that
there is something else that somebody _else_ can do in parallel
to make life easier for gitk and gitweb, and it is not on so
bare-metal-core side. The latest commit in Paul's "new" branch
says:
We were doing two execs for each tag - one to map the tag ID to a commit
ID (which git ls-remote was already giving us as refs/tags/foo^{}) and
one to read the contents of the tag for later display....
I think this is a typical problem any Porcelain layer faces. To
grab summary information for all refs in a concise form in one
go.
We discussed such a potential command about a month ago with
Jakub, but the suggestion did not go anywhere.
http://thread.gmane.org/gmane.comp.version-control.git/25013/focus=25055
In addition to what I described there, such a command should at
least allow specifying some special formatting for tag objects,
such as dereferencing it once (i.e. what the object name and
type of the referenced object is) and dereferencing it
repeatedly until a non-tag appears (i.e. "$that_tag^{}" along
with its type).
The show-ref command will alleviate the fork+exec problem; the
core side update you suggested for ref handling will improve the
performance of the implementation of such a command and can be
done in parallel.
^ permalink raw reply
* Re: Change set based shallow clone
From: Shawn Pearce @ 2006-09-11 6:08 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, Jakub Narebski
In-Reply-To: <7vac57nfb5.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano <junkio@cox.net> wrote:
> Shawn Pearce <spearce@spearce.org> writes:
> > This is probably something that is easily solved by a simple fixed
> > record format holding a 20 byte SHA1 (binary) and a fixed width null
> > terminated string holding the ref name, with the records sorted
> > by ref name. Its yet another file format with yet another set of
> > utilities needed but we pretty much have those (update-ref).
>
> Yup. That is one reasonable implementation of "single simple
> databasy file" I suggested. Or we could just borrow .git/info/refs
> format.
I think Linus was suggesting the same thing, just not nearly as
obvious. He also was looking for ways to not update it frequently.
But I can't say I'm a fan of negative entries. :-)
I'd probably rather code this as a fixed width binary heap file.
It should scale reasonable well to larger sets of refs. I know
someone on the list wanted to store Perforce change identifiers
as tags in Git but it was taking up more disk in ref files than
in pack-*.pack. That could be a very larger number of refs and
having O(log N) access there might be good.
But at this point I have more on my plate than I have time for. :-)
I've got to get that map window code merged against master or next
(preference Junio? next touches code I'm touching too but those
aren't in master yet) for 32 bit offset. Should be slightly easier
to do but the pack verification change really threw a wrench into
my merge. Other things have also kept me from really working on it.
I probably won't be able to look at it again until Wednesday.
I also want to spend more time on the dictionary packing prototype to
see if its worth spending even more time on. My amd64 box is happily
installing itself as write this (finally!) so I now have some more
reasonable hardware to deal with transforming the Mozilla pack. :)
--
Shawn.
^ permalink raw reply
* Re: [PATCH 0/2] Command preparation cleanup in "connect.c"
From: Junio C Hamano @ 2006-09-11 6:03 UTC (permalink / raw)
To: Christian Couder; +Cc: git
In-Reply-To: <200609110643.13486.chriscool@tuxfamily.org>
Christian Couder <chriscool@tuxfamily.org> writes:
> The following two patches clean "connect.c" up using "add_to_string"
> from "rsh.c". The code is still longer, but there is no leak and it may be
> better to die with an error message than to truncate a too long command.
Nice, clean, simple and obviously correct.
Thanks.
^ permalink raw reply
* Re: Change set based shallow clone
From: Junio C Hamano @ 2006-09-11 5:27 UTC (permalink / raw)
To: Shawn Pearce; +Cc: git, Jakub Narebski
In-Reply-To: <20060911024434.GA29368@spearce.org>
Shawn Pearce <spearce@spearce.org> writes:
> Jakub Narebski <jnareb@gmail.com> wrote:
>> Junio C Hamano wrote:
>>
>> > the impact of changing the
>> > one-file-per-ref implementation to something based on a single
>> > simple databasy file (e.g. gdbm, bdb, sqlite, ...)
>>
>> One of the complaints against Subversion was that it use BerkeleyDB
>> (bdb) backend... but it was before it acquired fsfs interface. Perhaps
>> we could use it too.
>
> I'm against the latest Berkely DB (Sleepy Cat) implementations.
> Every time I've stored data in them or in applications which use
> them I've lost everything. GNU dbm might not be too bad. SQL Lite
> is overkill.
>...
> This is probably something that is easily solved by a simple fixed
> record format holding a 20 byte SHA1 (binary) and a fixed width null
> terminated string holding the ref name, with the records sorted
> by ref name. Its yet another file format with yet another set of
> utilities needed but we pretty much have those (update-ref).
Yup. That is one reasonable implementation of "single simple
databasy file" I suggested. Or we could just borrow .git/info/refs
format.
^ permalink raw reply
* [PATCH 2/2] Fix a memory leak in "connect.c" and die if command too long.
From: Christian Couder @ 2006-09-11 5:04 UTC (permalink / raw)
To: Junio Hamano; +Cc: git
In-Reply-To: <20060911065922.93139e5d.chriscool@tuxfamily.org>
Use "add_to_string" instead of "sq_quote" and "snprintf", so
that there is no memory allocation and no memory leak.
Also check if the command is too long to fit into the buffer
and die if this is the case, instead of truncating it to the
buffer size.
Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---
connect.c | 17 ++++++++++++++---
1 files changed, 14 insertions(+), 3 deletions(-)
diff --git a/connect.c b/connect.c
index 1c6429b..49251f9 100644
--- a/connect.c
+++ b/connect.c
@@ -599,12 +599,13 @@ static void git_proxy_connect(int fd[2],
close(pipefd[1][0]);
}
+#define MAX_CMD_LEN 1024
+
/*
* Yeah, yeah, fixme. Need to pass in the heads etc.
*/
int git_connect(int fd[2], char *url, const char *prog)
{
- char command[1024];
char *host, *path = url;
char *end;
int c;
@@ -697,8 +698,18 @@ int git_connect(int fd[2], char *url, co
if (pid < 0)
die("unable to fork");
if (!pid) {
- snprintf(command, sizeof(command), "%s %s", prog,
- sq_quote(path));
+ char command[MAX_CMD_LEN];
+ char *posn = command;
+ int size = MAX_CMD_LEN;
+ int of = 0;
+
+ of |= add_to_string(&posn, &size, prog, 0);
+ of |= add_to_string(&posn, &size, " ", 0);
+ of |= add_to_string(&posn, &size, path, 1);
+
+ if (of)
+ die("command line too long");
+
dup2(pipefd[1][0], 0);
dup2(pipefd[0][1], 1);
close(pipefd[0][0]);
--
1.4.2.g430e
^ permalink raw reply related
* [PATCH 1/2] Move add_to_string to "quote.c" and make it extern.
From: Christian Couder @ 2006-09-11 4:59 UTC (permalink / raw)
To: Junio Hamano; +Cc: git
In-Reply-To: <200609110643.13486.chriscool@tuxfamily.org>
So that this function may be used in places other than "rsh.c".
Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---
quote.c | 29 +++++++++++++++++++++++++++++
quote.h | 6 ++++++
rsh.c | 29 -----------------------------
3 files changed, 35 insertions(+), 29 deletions(-)
diff --git a/quote.c b/quote.c
index a38786c..200e4e2 100644
--- a/quote.c
+++ b/quote.c
@@ -106,6 +106,35 @@ char *sq_quote_argv(const char** argv, i
return buf;
}
+/*
+ * Append a string to a string buffer, with or without shell quoting.
+ * Return true if the buffer overflowed.
+ */
+int add_to_string(char **ptrp, int *sizep, const char *str, int quote)
+{
+ char *p = *ptrp;
+ int size = *sizep;
+ int oc;
+ int err = 0;
+
+ if ( quote ) {
+ oc = sq_quote_buf(p, size, str);
+ } else {
+ oc = strlen(str);
+ memcpy(p, str, (oc >= size) ? size-1 : oc);
+ }
+
+ if ( oc >= size ) {
+ err = 1;
+ oc = size-1;
+ }
+
+ *ptrp += oc;
+ **ptrp = '\0';
+ *sizep -= oc;
+ return err;
+}
+
char *sq_dequote(char *arg)
{
char *dst = arg;
diff --git a/quote.h b/quote.h
index a6c4611..1a29e79 100644
--- a/quote.h
+++ b/quote.h
@@ -33,6 +33,12 @@ extern void sq_quote_print(FILE *stream,
extern size_t sq_quote_buf(char *dst, size_t n, const char *src);
extern char *sq_quote_argv(const char** argv, int count);
+/*
+ * Append a string to a string buffer, with or without shell quoting.
+ * Return true if the buffer overflowed.
+ */
+extern int add_to_string(char **ptrp, int *sizep, const char *str, int quote);
+
/* This unwraps what sq_quote() produces in place, but returns
* NULL if the input does not look like what sq_quote would have
* produced.
diff --git a/rsh.c b/rsh.c
index 07166ad..8a1db45 100644
--- a/rsh.c
+++ b/rsh.c
@@ -8,35 +8,6 @@ #include "cache.h"
#define COMMAND_SIZE 4096
-/*
- * Append a string to a string buffer, with or without shell quoting.
- * Return true if the buffer overflowed.
- */
-static int add_to_string(char **ptrp, int *sizep, const char *str, int quote)
-{
- char *p = *ptrp;
- int size = *sizep;
- int oc;
- int err = 0;
-
- if ( quote ) {
- oc = sq_quote_buf(p, size, str);
- } else {
- oc = strlen(str);
- memcpy(p, str, (oc >= size) ? size-1 : oc);
- }
-
- if ( oc >= size ) {
- err = 1;
- oc = size-1;
- }
-
- *ptrp += oc;
- **ptrp = '\0';
- *sizep -= oc;
- return err;
-}
-
int setup_connection(int *fd_in, int *fd_out, const char *remote_prog,
char *url, int rmt_argc, char **rmt_argv)
{
--
1.4.2.g430e
^ permalink raw reply related
* [PATCH 0/2] Command preparation cleanup in "connect.c" (was: [PATCH] Fix memory leak in "connect.c".)
From: Christian Couder @ 2006-09-11 4:43 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vwt8g48rt.fsf@assigned-by-dhcp.cox.net>
Le jeudi 7 septembre 2006 06:09, Junio C Hamano a écrit :
> Christian Couder <chriscool@tuxfamily.org> writes:
> > sq_quote allocates some memory that should be freed.
>
> That's technically correct, but the code you are touching is
> immediately before exec() or die() so I chose to be sloppy and
> short ;-).
>
> With sq_quote_buf() (which was introduced with 77d604 Oct 10
> 2005), you should be able to directly write into the command[]
> buffer, I think. The code you touched predates that function
> (b10d0e July 8 2005).
The following two patches clean "connect.c" up using "add_to_string"
from "rsh.c". The code is still longer, but there is no leak and it may be
better to die with an error message than to truncate a too long command.
Shortlog:
[PATCH 1/2] Move add_to_string to "quote.c" and make it extern.
[PATCH 2/2] Fix a memory leak in "connect.c" and die if command too
long.
Diffstat:
connect.c | 17 ++++++++++++++---
quote.c | 29 +++++++++++++++++++++++++++++
quote.h | 6 ++++++
rsh.c | 29 -----------------------------
4 files changed, 49 insertions(+), 32 deletions(-)
Christian.
^ permalink raw reply
* Re: Dropping Git.pm (at least Git.xs)?
From: Petr Baudis @ 2006-09-11 3:25 UTC (permalink / raw)
To: Sam Vilain, Jakub Narebski; +Cc: Dennis Stosberg, Junio C Hamano, git
In-Reply-To: <4504529A.70401@vilain.net>
Dear diary, on Sun, Sep 10, 2006 at 07:59:54PM CEST, I got a letter
where Sam Vilain <sam@vilain.net> said that...
> Dennis Stosberg wrote:
> > Having perl bindings to git internals and sometime in the future to a
> > libified git is a great thing. It will allow people to do interesting
> > things, quickly trying concepts without having to write any C code.
> > And I expect that gitweb can be sped up remarkably by using Git.pm (no
> > forking, parsing of command output often not necessary, easy caching of
> > frequently cached data across calls, etc)
>
> FWIW, I have been starting on a perl implementation. It uses the
> Git.pm, but not for anything *that* important. It's still very young,
> but once I have reading and writing files basically working, I'll
> release it to CPAN separately - no reason it needs to be distributed
> with Git itself.
>
> See http://utsl.gen.nz/gitweb/?p=VCS-Git
I think those two can coexist quite well. Yours aims for a nice and
slick object interface, while mine is just about wrapping Git interface
in Perl, without building any elaborate object model (it would provide
any only if underlying libgit would in the future, I guess).
Now, I'm not actually opposed to making Git.pm provide more
object-oriented interface, and it is thus obvious that it might be due
to consider a merge of the two modules, but
(i) Git.pm ought to stay bundled with Git, simply to be useful for Git
itself and the perl scripts it carries. Without Git.xs and associated
portability concerns, Git.pm might finally actually help things. :-)
(ii) People would thus go mad about external dependencies like Moose
or even Class::Autouse, so Git.pm can't rely on anything like that
(unless it bundles it, which is not quite practical in case of Moose).
(iii) (Besides, Moose may be the next totally cool and all-popular
thing in the world of Perl soon, but so far, I'd be personally careful
about using it for the "official" interface, since Git Perl developers
would apparently have to learn another way to do objects in Perl before
able to use Git, and other Perl developers going by couldn't read (and
fix/enhance) the code without doing the same.)
Dear diary, on Mon, Sep 11, 2006 at 12:13:01AM CEST, I got a letter
where Jakub Narebski <jnareb@gmail.com> said that...
> Could you please put appropriate information on GitWiki
> http://git.or.cz/gitwiki/InterfacesFrontendsAndTools
> Perhaps it would be good time to start new section, Git Implementations,
> and put egit (Java GIT library and Eclipse plugin) there too.
This really isn't a Git reimplementation (thankfully).
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Snow falling on Perl. White noise covering line noise.
Hides all the bugs too. -- J. Putnam
^ permalink raw reply
* Re: Change set based shallow clone
From: Linus Torvalds @ 2006-09-11 3:18 UTC (permalink / raw)
To: Paul Mackerras; +Cc: linux, jonsmirl, git
In-Reply-To: <Pine.LNX.4.64.0609101949580.27779@g5.osdl.org>
On Sun, 10 Sep 2006, Linus Torvalds wrote:
>
> If we did the same pack-file approach that we do for objects, the problem
> ends up being that _updating_ things is really hard. What we could do (and
> might work) is that a "git repack" would create a "packed representation
> of the heads too".
To clarify: I'm _not_ suggesting actually using the "pack-file"
representation itself for the references.
I'm saying that we could have something like a
.git/refs-packed
file, which could be (for example) just a plain linear text-file, of the
form
19ed368b17abfb9ad5c7467ea74fd8a045d96b43 refs/heads/html
60a6bf5f53635005f4f68d8b8a33172309193623 refs/heads/maint
2d32e76f893b2ac432201ce7596c8bab691691e6 refs/heads/man
a41fae9c46a4cb5e59cc1a17d19f6a3a6cbfbb2f refs/heads/master
61af0aaf26e003a61848e551bbd57e78e94eacdc refs/heads/next
585729203fef6ade64923277e7151b2e3a4ca330 refs/heads/pu
997283a8e87179b5b87a909686869d7843c8e19a refs/heads/todo
a0e7d36193b96f552073558acf5fcc1f10528917 refs/tags/junio-gpg-pub
d6602ec5194c87b0fc87103ca4d67251c76f233a refs/tags/v0.99
f25a265a342aed6041ab0cc484224d9ca54b6f41 refs/tags/v0.99.1
c5db5456ae3b0873fc659c19fafdde22313cc441 refs/tags/v0.99.2
7ceca275d047c90c0c7d5afb13ab97efdf51bd6e refs/tags/v0.99.3
b3e9704ecdf48869f635f0aa99ddfb513f885aff refs/tags/v0.99.4
07e38db6a5a03690034d27104401f6c8ea40f1fc refs/tags/v0.99.5
f12e22d4c12c3d0263fa681f25c06569f643da0f refs/tags/v0.99.6
f8696fcd2abc446a5ccda3e414b731eff2a7e981 refs/tags/v0.99.7
1094cf40f7029f803421c1dcc971238507c830c5 refs/tags/v0.99.7a
da30c6c39cd3b048952a15929c5440acfd71b912 refs/tags/v0.99.7b
9165ec17fde255a1770886189359897dbb541012 refs/tags/v0.99.7c
02b2acff8bafb6d73c6513469cdda0c6c18c4138 refs/tags/v0.99.7d
...
ie it would contain just a linear file with the "<hex></tab><refname>"
format. Then, the way to look up a reference would be:
- look it up in the traditional loose file
- if it exists, and contains zeros (or not a hex value), it's considered
a "negative entry", and the branch doesn't exist
- otherwise, if it's a good SHA1, that's the result
- if it's not there, look it up in the ".git/refs-packed" file by just
doing a simple linear scan (trivial, and actually efficient - we're
talking about just a few kB of memory after all, and the _cost_ is
actually the IO, where "simple linear scan" is actually very good for
performance).
The end result would be that we'd probably have very few loose references
(we'd get them whenever we change a ref, or delete one), making the lookup
scale better. The big _bulk_ of the references tend to be very stable,
notably they are tags that seldom - if ever - change, and would thus stay
just in the packed refs file.
So the normal situation would be that you'd have a few hundred (maybe, for
a bigger project) refs in the single .git/refs-packed file, totalling a
few kB of disk-space, and then you might have a handful of "active" heads
that are in the traditional single-file format .git/refs/<filename>
because they are beign actively modified and have changed since the last
repack.
I bet it would work fairly well. But somebody would need to implement it.
The good news is that the refs-handling code tends to be _fairly_ well
abstracted out, because we already wanted that for the logging thing. So
we hopefully already don't actually access the loose file objects by hand
from shell scripts any more - we use git-rev-parse and git-update-ref etc
to look up refnames, and that all goes back to git/refs.c.
So _most_ of the bulk of it would probably be in refs.c, but there's
obviously also things like git-branch.sh that needs to be taught the new
rules about deleting branches etc.
Anybody want to try the above rules out? I bet the _only_ real issue would
be "for_each_ref()", where it's important to _first_ do the loose objects,
and remember them all, so that you do _not_ show the refs that are in the
.git/refs-packed file and already got shown because they were loose.
NOTE! It's important that whatever sceme used gets locking right. The
above suggestion gets it right simply because it doesn't really _change_
anything. Any new or modified ref ends up using the old code, and using a
".lock" file and renaming it automatically does the same thing it ever
did.
Linus
^ permalink raw reply
* Re: Change set based shallow clone
From: Linus Torvalds @ 2006-09-11 2:55 UTC (permalink / raw)
To: Paul Mackerras; +Cc: linux, jonsmirl, git
In-Reply-To: <17668.38032.20688.991295@cargo.ozlabs.ibm.com>
On Mon, 11 Sep 2006, Paul Mackerras wrote:
>
> Do you think there is any way to speed up the cold-cache case for
> git-ls-remote and git-rev-parse with thousands of heads and tags?
Nothing obvious comes to mind.
If we did the same pack-file approach that we do for objects, the problem
ends up being that _updating_ things is really hard. What we could do (and
might work) is that a "git repack" would create a "packed representation
of the heads too".
The issue is that once you create a pack, you _really_ don't want to use
read-modify-write to modify it ever afterwards. That's how you get nasty
corruption. The "append-only" side of the git object database is really
what makes things so stable, and packing multiple heads in the same file
automatically means that if something goes wrong, it's _disastrously_
wrong, in a way it isn't when you have separate files.
So we could generate a "pack of references", but then any modifications
would be done with the current loose "file objects" approach, and just
have the filesystem override the pack-files. The problem then actually
becomes one of _deleting_ branches, because then we'd have to add a
"negative branch" loose object. Ugly.
An alternate approach might be to keep the current filesystem layour, but
simply do the readdir over the whole reference situation in one pass, then
sort them by inode number, and look them up in order. That would help on
some filesystems, at least.
Linus
^ permalink raw reply
* Re: Change set based shallow clone
From: Linus Torvalds @ 2006-09-11 2:49 UTC (permalink / raw)
To: Jon Smirl; +Cc: linux, git, paulus
In-Reply-To: <9e4733910609101400j2604ace1vdb2e363c1296418f@mail.gmail.com>
On Sun, 10 Sep 2006, Jon Smirl wrote:
>
> Is the kernel mapped into user processes using huge pages?
Yup. Huge pages and global. So the kernel should have fairly low TLB
impact (depending on CPU - the kernel also has less locality than user
space, since the data structures are all over the place, and some CPU's
have very few hugepage entries. System code generally tends to have much
worse I$ and TLB behaviour than most user apps, but at least the hugepage
mappings tend to mitigate the TLB side somewhat).
> Another thing that should be mapped with huge pages is the video RAM.
I doubt it ends up being a huge deal in most cases. If you don't use the
graphics chip to access video RAM with accelerated methods, the TLB is the
_least_ of your worries. Going over the PCI bus (even with PCI-X etc) is
the real problem.
There's a reason you want to do acceleration even in 2D..
Linus
^ permalink raw reply
* Re: Change set based shallow clone
From: Shawn Pearce @ 2006-09-11 2:44 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <ee2cms$o18$1@sea.gmane.org>
Jakub Narebski <jnareb@gmail.com> wrote:
> Junio C Hamano wrote:
>
> > the impact of changing the
> > one-file-per-ref implementation to something based on a single
> > simple databasy file (e.g. gdbm, bdb, sqlite, ...)
>
> One of the complaints against Subversion was that it use BerkeleyDB
> (bdb) backend... but it was before it acquired fsfs interface. Perhaps
> we could use it too.
I'm against the latest Berkely DB (Sleepy Cat) implementations.
Every time I've stored data in them or in applications which use
them I've lost everything. GNU dbm might not be too bad. SQL Lite
is overkill.
I was thinking of using a tree object only because its a well
defined Git file format that's already in use. Plus its "easy"
to repair by loading it into an index, twiddline it and invoking
git-write-tree to convert it back. But there's a lot of downsides.
This is probably something that is easily solved by a simple fixed
record format holding a 20 byte SHA1 (binary) and a fixed width null
terminated string holding the ref name, with the records sorted
by ref name. Its yet another file format with yet another set of
utilities needed but we pretty much have those (update-ref).
> Or perhaps it is for something like Electra, or ReiserFS file properites
> access...
Except not everyone has those filesystems. :-)
--
Shawn.
^ permalink raw reply
* What's in git.git
From: Junio C Hamano @ 2006-09-11 2:21 UTC (permalink / raw)
To: git
It's been two weeks since the last "What's in" update, so here
is the current status.
* The 'master' branch has these since the last announcement.
- Andy Whitcroft spotted a long-standing bug that prevented
send-pack to deal correctly with a ref whose name is longer
than 45 bytes, where we did not have to have any such limit.
- Jakub Narebski keeps working on gitweb, with help from Aneesh
Kumar, Dennis Stosberg, Luben Tuikov, and Martin Waitz.
There are a lot of clean-ups, including these notables:
- mechanism to selectively enable or disable features by site
administrators and repository owners.
- snapshot and blame are now elective features using the above.
- gitweb no longer uses temporary files to generate diffs.
- Jakub also updated a few autoconf stuff.
- Christian Couder's GIT_TRACE updates.
- Franck Bui-Huu's clean-up to the code for "format-patch -s".
- git-daemon acquired a mechanism to selectively enable or
disable features by site administrators and repository
owners.
- pack-objects validates the data it copies from existing pack
or new-style loose objects.
- Other small clean-ups, fixes and updates from Johannes Schindelin,
Jonas Fonseca, Linus Torvalds, Martin Langhoff, Matthias Kestenholz,
Sergey Vlasov and Shawn Pearce.
- gitk updates from Paul Mackerras.
* The 'next' branch, in addition, has these.
- Andy Whitcroft taught send-pack to use git-rev-list --stdin
so that we do not have to be limited by the number of refs
exec() command-line can hold.
- Pasky's Git.pm is on hold; it was discussed and agreed that
Git.xs layer was a bit premature and is hurting the adoption
of the entire series.
- Franck Bui-Huu and Rene Scharfe with a bit help from me added
git-archive command to unify git-tar-tree/git-zip-tree and
make them accessible over network.
- Jeff King rewrote run_status() shell function in git-commit
and git-status in C.
- Per requests from the list, "git apply" automatically applies
binary patches without having to be given --binary flag.
- Likewise, "git diff --binary" does not give full index line for
non-binary part of the patch anymore.
- Pack-objects learned to run rev-list logic internally when
given --revs parameter; the refs arguments you would normally
give the upstream rev-list can be fed from its standard
input, instead of usual list of objects.
- Pack-objects also knows how to pretend objects that are in
named packs are unpacked. This would make easy to update
repack to incrementally pack loose objects and recent
"active" pack(s).
- I have a few patches to upload-pack that would help
upload-pack when downloader has more roots than the uploader
has, but this is frozen until I hear real-world feedback.
- unpack-objects learned a trick not to stop when fed a corrupt
pack; instead it can make the best effort to recover from
such an error that was detected.
* The 'pu' branch, in addition, has these.
- I have a wip to implement index, working tree and zero or
more trees in parallel but I haven't looked at it for some
time.
^ permalink raw reply
* Re: Change set based shallow clone
From: Jon Smirl @ 2006-09-11 2:11 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Shawn Pearce, git
In-Reply-To: <7vfyezqlny.fsf@assigned-by-dhcp.cox.net>
On 9/10/06, Junio C Hamano <junkio@cox.net> wrote:
> I do not see any advantage of making such a phoney object at
> all, but I do agree that the current one file per ref can be
> less than optimal when your repository has tons of tags or
> refs.
Here's a hack, instead of of putting the sha inside the file, put the
sha into the filename.
master_86a8534ba23a5532f6d0ddd01ecd8f02f662cf78
Now you can just do a directory listing and get all of the data
quickly. To keep the existing porcelain working add a symlink.
ln -s master_86a8534ba23a5532f6d0ddd01ecd8f02f662cf78 master
You might want the sha1 encoded names in a new directory
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: Change set based shallow clone
From: Jakub Narebski @ 2006-09-11 1:04 UTC (permalink / raw)
To: git
In-Reply-To: <7vfyezqlny.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano wrote:
> the impact of changing the
> one-file-per-ref implementation to something based on a single
> simple databasy file (e.g. gdbm, bdb, sqlite, ...)
One of the complaints against Subversion was that it use BerkeleyDB
(bdb) backend... but it was before it acquired fsfs interface. Perhaps
we could use it too.
Or perhaps it is for something like Electra, or ReiserFS file properites
access...
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: Change set based shallow clone
From: Junio C Hamano @ 2006-09-11 0:42 UTC (permalink / raw)
To: Shawn Pearce; +Cc: git
In-Reply-To: <20060911000403.GB28927@spearce.org>
Shawn Pearce <spearce@spearce.org> writes:
> Junio C Hamano <junkio@cox.net> wrote:
>> We could benefit from the suggested organization of one base
>> archive pack plus a current active pack. The core side code to
>> help doing so was posted here which followed a discussion on how
>> to have repack make use of it last week.
>>
>> http://thread.gmane.org/gmane.comp.version-control.git/26218/focus=26326
>>
>> Any takers?
>
> I thought this was settled and you had it implemented? Is this
> still an open topic?
I've implemented the core side and I think it is ready. People
who actually have need for such a repack are welcome to take a
crack at it at the Porcelain-ish level.
^ permalink raw reply
* Re: Change set based shallow clone
From: Junio C Hamano @ 2006-09-11 0:41 UTC (permalink / raw)
To: Shawn Pearce; +Cc: git
In-Reply-To: <20060911000306.GA28927@spearce.org>
Shawn Pearce <spearce@spearce.org> writes:
> I've been thinking about implementing ref storage within a Git tree
> object. Just store the commit/tag/object IDs in a tree (or graph
> of trees) with a mode of '0'. Anchor that under '.git/refs-tree'.
> Any edit of a ref would "lock" .git/refs-tree, create a new tree
> containing the update, then replace .git/refs-tree.
>
> But it would put additional stress on the objects directory by
> creating a lot of trees which would never get pulled into pack
> files and thus would need to be pruned away on a regular basis.
I do not see any advantage of making such a phoney object at
all, but I do agree that the current one file per ref can be
less than optimal when your repository has tons of tags or
refs.
We've designed the ref interface well enough so that only very
narrow core functions know the implementation (and properly
written Porcelain would access refs via update-ref and
symbolic-ref interfaces), so the impact of changing the
one-file-per-ref implementation to something based on a single
simple databasy file (e.g. gdbm, bdb, sqlite, ...) would be
reasonably limited.
^ permalink raw reply
* Re: Change set based shallow clone
From: Shawn Pearce @ 2006-09-11 0:04 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Jon Smirl, git
In-Reply-To: <7v1wqjsgfx.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano <junkio@cox.net> wrote:
> We could benefit from the suggested organization of one base
> archive pack plus a current active pack. The core side code to
> help doing so was posted here which followed a discussion on how
> to have repack make use of it last week.
>
> http://thread.gmane.org/gmane.comp.version-control.git/26218/focus=26326
>
> Any takers?
I thought this was settled and you had it implemented? Is this
still an open topic?
--
Shawn.
^ permalink raw reply
* Re: Change set based shallow clone
From: Shawn Pearce @ 2006-09-11 0:03 UTC (permalink / raw)
To: Paul Mackerras
Cc: Jon Smirl, Linus Torvalds, linux@horizon.com, Git Mailing List
In-Reply-To: <17668.2019.732961.855446@cargo.ozlabs.ibm.com>
Paul Mackerras <paulus@samba.org> wrote:
[snip]
> The bottom line is that I can speed up the startup for the hot-cache
> case quite a lot. The cold-cache case is going to take about 20-30
> seconds whatever I do unless Linus or Junio can come up with a way to
> pack the heads and tags. I could read the refs asynchronously but
> there will still be a long delay in git rev-parse if you give
> arguments such as --all.
I've been thinking about implementing ref storage within a Git tree
object. Just store the commit/tag/object IDs in a tree (or graph
of trees) with a mode of '0'. Anchor that under '.git/refs-tree'.
Any edit of a ref would "lock" .git/refs-tree, create a new tree
containing the update, then replace .git/refs-tree.
But it would put additional stress on the objects directory by
creating a lot of trees which would never get pulled into pack
files and thus would need to be pruned away on a regular basis.
It also would make parallel updates more difficult on the server
side as everyone would need to wait for the lock to .git/refs-tree
before they can change any ref; today users only need to wait for
the ref they are trying to change.
It also doesn't help looking up a ref quickly; although trees are
sorted they are variable length entries which forces the application
to read the entire tree to find its entry.
Given those three downsides I haven't put anything to code yet.
--
Shawn.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox