* Re: Add specialized object allocator
From: Linus Torvalds @ 2006-06-19 18:10 UTC (permalink / raw)
To: Junio C Hamano, Git Mailing List
In-Reply-To: <Pine.LNX.4.64.0606191028540.5498@g5.osdl.org>
On Mon, 19 Jun 2006, Linus Torvalds wrote:
>
> It also allows us to track some basic statistics about object
> allocations. For example, for the mozilla import, it shows
> object usage as follows:
>
> blobs: 627629 (14710 kB)
> trees: 1119035 (34969 kB)
> commits: 196423 (8440 kB)
> tags: 1336 (46 kB)
Btw, this is the trivial additional patch to allow you to say
git --report rev-list --all --objects > /dev/null
and have it report object allocation usage after the op.
Useful? Probably not. It was useful for me to verify that everything
looked ok (and while I knew we had more trees than blobs, it's actually
interesting to see how projects differ.
The "git" tree, for example, has more blobs than trees: because it's a
fairly flat development tree, a commit that changes more than one file
will generate more new blobs than it generates trees.
In contrast, the kernel has about 30% more trees than blobs (but since
trees have the extra data/size fields, 30% extra trees take 75% more space
than blobs).
The mozilla import has a _lot_ more trees than blobs (80% more trees, and
they use up more than twice as much memory). It's probably a pretty deep
tree structure and/or they often commit changes to single files deep into
the tree).
Linus
----
diff --git a/git.c b/git.c
index 329ebec..6149499 100644
--- a/git.c
+++ b/git.c
@@ -271,6 +271,10 @@ int main(int argc, const char **argv, ch
puts(git_exec_path());
exit(0);
}
+ if (!strcmp(cmd, "report")) {
+ atexit(alloc_report);
+ continue;
+ }
cmd_usage(0, NULL, NULL);
}
argv[0] = cmd;
^ permalink raw reply related
* Add specialized object allocator
From: Linus Torvalds @ 2006-06-19 17:44 UTC (permalink / raw)
To: Junio C Hamano, Git Mailing List
This creates a simple specialized object allocator for basic
objects.
This avoids wasting space with malloc overhead (metadata and
extra alignment), since the specialized allocator knows the
alignment, and that objects, once allocated, are never freed.
It also allows us to track some basic statistics about object
allocations. For example, for the mozilla import, it shows
object usage as follows:
blobs: 627629 (14710 kB)
trees: 1119035 (34969 kB)
commits: 196423 (8440 kB)
tags: 1336 (46 kB)
and the simpler allocator shaves off about 2.5% off the memory
footprint off a "git-rev-list --all --objects", and is a bit
faster too.
[ Side note: this concludes the series of "save memory in object storage".
The thing is, there simply isn't much more to be saved on the objects.
Doing "git-rev-list --all --objects" on the mozilla archive has a final
total RSS of 131498 pages for me: that's about 513MB. Of that, the
object overhead is now just 56MB, the rest is going somewhere else (put
another way: the fact that this patch shaves off 2.5% of the total
memory overhead, considering that objects are now not much more than 10%
of the total shows how big the wasted space really was: this makes
object allocations much more memory- and time-efficient).
I haven't looked at where the rest is, but I suspect the bulk of it is
just the pack-file loading. It may be that we should pack the tree
objects separately from the blob objects: for git-rev-list --objects, we
don't actually ever need to even look at the blobs, but since trees and
blobs are interspersed in the pack-file, we end up not being dense in
the tree accesses, so we end up looking at more pages than we strictly
need to.
So with a 535MB pack-file, it's entirely possible - even likely - that
most of the remaining RSS is just the mmap of the pack-file itself. We
don't need to map in _all_ of it, but we do end up mapping a fair
amount. ]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
---
Makefile | 2 +-
alloc.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++
blob.c | 2 +-
cache.h | 11 +++++++++++
commit.c | 2 +-
tag.c | 2 +-
tree.c | 2 +-
7 files changed, 67 insertions(+), 5 deletions(-)
diff --git a/Makefile b/Makefile
index ea8cd28..0887945 100644
--- a/Makefile
+++ b/Makefile
@@ -216,7 +216,7 @@ LIB_OBJS = \
server-info.o setup.o sha1_file.o sha1_name.o strbuf.o \
tag.o tree.o usage.o config.o environment.o ctype.o copy.o \
fetch-clone.o revision.o pager.o tree-walk.o xdiff-interface.o \
- $(DIFF_OBJS)
+ alloc.o $(DIFF_OBJS)
BUILTIN_OBJS = \
builtin-log.o builtin-help.o builtin-count.o builtin-diff.o builtin-push.o \
diff --git a/alloc.c b/alloc.c
new file mode 100644
index 0000000..65ec0cc
--- /dev/null
+++ b/alloc.c
@@ -0,0 +1,51 @@
+/*
+ * alloc.c - specialized allocator for internal objects
+ *
+ * Copyright (C) 2006 Linus Torvalds
+ *
+ * The standard malloc/free wastes too much space for objects, partly because
+ * it maintains all the allocation infrastructure (which isn't needed, since
+ * we never free an object descriptor anyway), but even more because it ends
+ * up with maximal alignment because it doesn't know what the object alignment
+ * for the new allocation is.
+ */
+#include "cache.h"
+#include "object.h"
+#include "blob.h"
+#include "tree.h"
+#include "commit.h"
+#include "tag.h"
+
+#define BLOCKING 1024
+
+#define DEFINE_ALLOCATOR(name) \
+static unsigned int name##_allocs; \
+struct name *alloc_##name##_node(void) \
+{ \
+ static int nr; \
+ static struct name *block; \
+ \
+ if (!nr) { \
+ nr = BLOCKING; \
+ block = xcalloc(BLOCKING, sizeof(struct name)); \
+ } \
+ nr--; \
+ name##_allocs++; \
+ return block++; \
+}
+
+DEFINE_ALLOCATOR(blob);
+DEFINE_ALLOCATOR(tree);
+DEFINE_ALLOCATOR(commit);
+DEFINE_ALLOCATOR(tag);
+
+#define REPORT(name) \
+ fprintf(stderr, "%10s: %8u (%zu kB)\n", #name, name##_allocs, name##_allocs*sizeof(struct name) >> 10)
+
+void alloc_report(void)
+{
+ REPORT(blob);
+ REPORT(tree);
+ REPORT(commit);
+ REPORT(tag);
+}
diff --git a/blob.c b/blob.c
index 7377008..496f270 100644
--- a/blob.c
+++ b/blob.c
@@ -8,7 +8,7 @@ struct blob *lookup_blob(const unsigned
{
struct object *obj = lookup_object(sha1);
if (!obj) {
- struct blob *ret = xcalloc(1, sizeof(struct blob));
+ struct blob *ret = alloc_blob_node();
created_object(sha1, &ret->object);
ret->object.type = TYPE_BLOB;
return ret;
diff --git a/cache.h b/cache.h
index 7fcb6d4..eaa5c0c 100644
--- a/cache.h
+++ b/cache.h
@@ -384,4 +384,15 @@ extern void setup_pager(void);
int decode_85(char *dst, char *line, int linelen);
void encode_85(char *buf, unsigned char *data, int bytes);
+/* alloc.c */
+struct blob;
+struct tree;
+struct commit;
+struct tag;
+extern struct blob *alloc_blob_node(void);
+extern struct tree *alloc_tree_node(void);
+extern struct commit *alloc_commit_node(void);
+extern struct tag *alloc_tag_node(void);
+extern void alloc_report(void);
+
#endif /* CACHE_H */
diff --git a/commit.c b/commit.c
index 5914200..0fa1198 100644
--- a/commit.c
+++ b/commit.c
@@ -84,7 +84,7 @@ struct commit *lookup_commit(const unsig
{
struct object *obj = lookup_object(sha1);
if (!obj) {
- struct commit *ret = xcalloc(1, sizeof(struct commit));
+ struct commit *ret = alloc_commit_node();
created_object(sha1, &ret->object);
ret->object.type = TYPE_COMMIT;
return ret;
diff --git a/tag.c b/tag.c
index 9191333..5f70a5b 100644
--- a/tag.c
+++ b/tag.c
@@ -19,7 +19,7 @@ struct tag *lookup_tag(const unsigned ch
{
struct object *obj = lookup_object(sha1);
if (!obj) {
- struct tag *ret = xcalloc(1, sizeof(struct tag));
+ struct tag *ret = alloc_tag_node();
created_object(sha1, &ret->object);
ret->object.type = TYPE_TAG;
return ret;
diff --git a/tree.c b/tree.c
index 64422fd..1023655 100644
--- a/tree.c
+++ b/tree.c
@@ -129,7 +129,7 @@ struct tree *lookup_tree(const unsigned
{
struct object *obj = lookup_object(sha1);
if (!obj) {
- struct tree *ret = xcalloc(1, sizeof(struct tree));
+ struct tree *ret = alloc_tree_node();
created_object(sha1, &ret->object);
ret->object.type = TYPE_TREE;
return ret;
^ permalink raw reply related
* Re: [PATCH] cvsimport: ignore CVSPS_NO_BRANCH and impossible branches
From: Salikh Zakirov @ 2006-06-19 16:20 UTC (permalink / raw)
Cc: git
In-Reply-To: <7vzmgb8plx.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano wrote:
> Martin Langhoff <martin@catalyst.net.nz> writes:
>
>> cvsps output often contains references to CVSPS_NO_BRANCH, commits that it
>> could not trace to a branch. Ignore that branch.
>>
>> Additionally, cvsps will sometimes draw circular relationships between
>> branches -- where two branches are recorded as opening from the other.
>> In those cases, and where the ancestor branch hasn't been seen, ignore
>> it.
>
> This sounds more like an workaround than a real fix to me,
> although I'd apply it for now. I see Yann is collecting cvsps
> patches but maybe there will be a real fix soonish?
#CVSPS_NO_BRANCH is the identifier that CVSPS gives to the unnamed branches.
Unnamed branches appear when the branch tag is removed or moved forcefully.
The following short script reproduces the CVS repository with unnamed branches.
In my opinion, ignoring #CVSPS_NO_BRANCH in git-cvsimport is the most sensible
thing to do, because the branch was abandoned in the CVS in the first place.
I had to preprocess cvsps output to cut out CVSPS_NO_BRANCH commits anyway.
I vote for including the Martin's patch.
Creating CVS repository with unnamed branch
--8<--
export CVSROOT=$PWD/cvsroot
cvs init
cvs checkout .
mkdir a
cvs add a
cd a
vim a.txt
cvs add a.txt
cvs commit -m "added a.txt" a.txt
cvs tag -b br1
cvs update -r br1
echo "br1 update" >> a.txt
cvs commit -m "br1 update" a.txt
cvs update -A
echo "HEAD update" >> a.txt
cvs commit -m "HEAD update" a.txt
cvs tag -d br1
cvs tag -b br1
cvs tag -d -B br1
cvs tag -b br1
cvs update -r br1
echo "branch update, once more" >> a.txt
cvs commit -m "2nd branch update" a.txt
------
and corresponding CVSPS output is
--8<--
$ cvsps -A
WARNING: revision 1.1.2.1 of file a.txt on unnamed branch
---------------------
PatchSet 1
Date: 2006/06/19 20:10:09
Author: sszakiro
Branch: HEAD
Tag: (none)
Log:
added a.txt
Members:
a.txt:INITIAL->1.1
---------------------
PatchSet 2
Date: 2006/06/19 20:10:45
Author: sszakiro
Branch: #CVSPS_NO_BRANCH
Ancestor branch: HEAD
Tag: (none)
Log:
br1 update
Members:
a.txt:1.1->1.1.2.1
---------------------
PatchSet 3
Date: 2006/06/19 20:11:18
Author: sszakiro
Branch: HEAD
Tag: (none)
Log:
HEAD update
Members:
a.txt:1.1->1.2
---------------------
PatchSet 4
Date: 2006/06/19 20:12:07
Author: sszakiro
Branch: br1
Ancestor branch: HEAD
Tag: (none)
Log:
2nd branch update
Members:
a.txt:1.2->1.2.2.1
------
^ permalink raw reply
* Re: [PATCH] Make t8001-annotate and t8002-blame more portable
From: Alex Riesen @ 2006-06-19 16:35 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Dennis Stosberg, git
In-Reply-To: <7v3be218ri.fsf@assigned-by-dhcp.cox.net>
On 6/18/06, Junio C Hamano <junkio@cox.net> wrote:
> > + 'perl -pi -e "s/^1A.*\n$//; s/^3A/99/" file &&
activestate perl again: doesn't work without giving a backup file extension,
like this: perl -pi.bak -e ...
^ permalink raw reply
* Re: simple use case scenario for --read-tree and --write-tree with --prefix option
From: Aneesh Kumar @ 2006-06-19 15:50 UTC (permalink / raw)
To: Petr Baudis; +Cc: Junio C Hamano, git
In-Reply-To: <20060619135343.GP2609@pasky.or.cz>
On 6/19/06, Petr Baudis <pasky@suse.cz> wrote:
> Dear diary, on Mon, Jun 19, 2006 at 09:58:31AM CEST, I got a letter
> where Aneesh Kumar <aneesh.kumar@gmail.com> said that...
> > cd test
> >
> > kvaneesh@satan:/tmp/test$ git read-tree --prefix=test1/ $(cat
> > test1/.git/refs/heads/master)
> > fatal: failed to unpack tree object c6c049d03f0bee0ac546ff6e436d5f6f3a5f4864
> >
> > But the above command doesn't work for me. I guess i am missing something.
>
> You need to do a fetch to your /tmp/test repository first.
>
Ok that made the --read-tree part work. But the log or whatchanged
commands doesn't gets me the history of changes with resepect to the
files added. Now i remember this is in a way simillar to gitweb
history not shown. But then using the --full-history option with git
log also doesn't show the details of history with respect to the
files.
This is what i did
git fetch ./test2/
git read-tree --prefix=test2/ $(cat test2/.git/refs/heads/master)
Now if i look at .git/objects/ I can see commit objects with respect
to changes for test2/a
-aneesh
^ permalink raw reply
* Re: simple use case scenario for --read-tree and --write-tree with --prefix option
From: Petr Baudis @ 2006-06-19 13:53 UTC (permalink / raw)
To: Aneesh Kumar; +Cc: Junio C Hamano, git
In-Reply-To: <cc723f590606190058w2d7481ecsaded46095aee2355@mail.gmail.com>
Dear diary, on Mon, Jun 19, 2006 at 09:58:31AM CEST, I got a letter
where Aneesh Kumar <aneesh.kumar@gmail.com> said that...
> cd test
>
> kvaneesh@satan:/tmp/test$ git read-tree --prefix=test1/ $(cat
> test1/.git/refs/heads/master)
> fatal: failed to unpack tree object c6c049d03f0bee0ac546ff6e436d5f6f3a5f4864
>
> But the above command doesn't work for me. I guess i am missing something.
You need to do a fetch to your /tmp/test repository first.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
A person is just about as big as the things that make them angry.
^ permalink raw reply
* Re: git-rebase nukes multiline comments
From: Matthias Hopf @ 2006-06-19 9:54 UTC (permalink / raw)
To: git, xorg
In-Reply-To: <20060619093623.GA15209@suse.de>
On Jun 19, 06 11:36:23 +0200, Matthias Hopf wrote:
> On Jun 16, 06 12:55:43 -0500, David Kowis wrote:
> O-key. Did this work w/o a blank line as well? Then we can assume this
Ignore my ignorance. I really should *read* before answering...
Sorry
Matthias
--
Matthias Hopf <mhopf@suse.de> __ __ __
Maxfeldstr. 5 / 90409 Nuernberg (_ | | (_ |__ mat@mshopf.de
Phone +49-911-74053-715 __) |_| __) |__ labs www.mshopf.de
^ permalink raw reply
* Re: git-rebase nukes multiline comments
From: Matthias Hopf @ 2006-06-19 9:53 UTC (permalink / raw)
To: git
In-Reply-To: <7vwtbgbsax.fsf@assigned-by-dhcp.cox.net>
On Jun 16, 06 16:21:26 -0700, Junio C Hamano wrote:
> Having said that, I would say it is a bug. We should be able to
> rebase, cherry-pick and/or rebase a patch with an arbitrary
> binary garbage in the commit log message (I think the latter two
> command do). But because of the reason (2) above, it is very
> low on my priority to change it.
I understand. Many thanks for your explainations. I think this intended
log format should be documented somewhere, that would help a lot. I
don't think that many developers using git in CVS style know about this
convention.
Said that, I assume git nuking the multiline comment was a bug in 1.3.1
that has been (somewhat ;) solved.
Matthias
--
Matthias Hopf <mhopf@suse.de> __ __ __
Maxfeldstr. 5 / 90409 Nuernberg (_ | | (_ |__ mat@mshopf.de
Phone +49-911-74053-715 __) |_| __) |__ labs www.mshopf.de
^ permalink raw reply
* Re: git-rebase nukes multiline comments
From: Matthias Hopf @ 2006-06-19 9:36 UTC (permalink / raw)
To: git; +Cc: xorg
In-Reply-To: <4492F09F.9080906@shlrm.org>
On Jun 16, 06 12:55:43 -0500, David Kowis wrote:
> > I'm new to git, but I tried what you said.
> > my git log:
> > commit c846bea8c61bec7cf0f7688c48abc42577b9ac7f
> > Author: David Kowis <dkowis@kain.org>
> > Date: Fri Jun 16 12:20:08 2006 -0500
> >
> > this is a multi
> >
> > line comment
> > with three lines
> >
> >
> > I'm using git 1.4.0. It added a blank line in there...
O-key. Did this work w/o a blank line as well? Then we can assume this
solved in 1.4.0. Now there's still the question whether the log messages
in the upstream archive can be restored...
> I'm going to note that the xorg ML cc doesn't work for anyone not
> subscribed... You may miss out on replies.
I'm subscribed here as well :-)
I just CC'ed xorg so people over there know about the issue as well.
CU
Matthias
--
Matthias Hopf <mhopf@suse.de> __ __ __
Maxfeldstr. 5 / 90409 Nuernberg (_ | | (_ |__ mat@mshopf.de
Phone +49-911-74053-715 __) |_| __) |__ labs www.mshopf.de
^ permalink raw reply
* Re: [PATCH 1/7] Remove ranges from switch statements.
From: linux @ 2006-06-19 9:17 UTC (permalink / raw)
To: git, octo, tihirvon
> Yes, but isalnum(ch) even better ;)
And if you want to get fancy, there's plenty of room in git's sane_ctype
array for an additional punctuation bit to catch the [./-] cases.
Note also that technically the set of unreserved characters in URIs
(according to rfc2396 section 2.3) is
unreserved = alphanum | mark
mark = "-" | "_" | "." | "!" | "~" | "*" | "'" | "(" | ")"
The complementary set of characters that should be quoted is defined as well:
(2.2)
Many URI include components consisting of or delimited by, certain
special characters. These characters are called "reserved", since
their usage within the URI component is limited to their reserved
purpose. If the data for a URI component would conflict with the
reserved purpose, then the conflicting data must be escaped before
forming the URI.
reserved = ";" | "/" | "?" | ":" | "@" | "&" | "=" | "+" |
"$" | ","
(2.4.3)
The angle-bracket "<" and ">" and double-quote (") characters are
excluded because they are often used as the delimiters around URI in
text documents and protocol fields. The character "#" is excluded
because it is used to delimit a URI from a fragment identifier in URI
references (Section 4). The percent character "%" is excluded because
it is used for the encoding of escaped characters.
delims = "<" | ">" | "#" | "%" | <">
Other characters are excluded because gateways and other transport
agents are known to sometimes modify such characters, or they are
used as delimiters.
unwise = "{" | "}" | "|" | "\" | "^" | "[" | "]" | "`"
E.g. (the following change is placed in the public domain):
diff --git a/ctype.c b/ctype.c
index 56bdffa..01724d1 100644
--- a/ctype.c
+++ b/ctype.c
@@ -8,16 +8,17 @@ #include "cache.h"
#define SS GIT_SPACE
#define AA GIT_ALPHA
#define DD GIT_DIGIT
+#define MM GIT_URIMARK /* Legal URI "marks", RFC2396 section 2.3, plus / */
unsigned char sane_ctype[256] = {
0, 0, 0, 0, 0, 0, 0, 0, 0, SS, SS, 0, 0, SS, 0, 0, /* 0-15 */
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 16-15 */
- SS, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 32-15 */
+ SS, MM, 0, 0, 0, 0, 0, MM, MM, MM, MM, 0, 0, MM, MM, MM, /* 32-15 */
DD, DD, DD, DD, DD, DD, DD, DD, DD, DD, 0, 0, 0, 0, 0, 0, /* 48-15 */
0, AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, /* 64-15 */
- AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, 0, 0, 0, 0, 0, /* 80-15 */
+ AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, 0, 0, 0, 0, MM, /* 80-15 */
0, AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, /* 96-15 */
- AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, 0, 0, 0, 0, 0, /* 112-15 */
+ AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, AA, 0, 0, 0, MM, 0, /* 112-15 */
/* Nothing in the 128.. range */
};
diff --git a/git-compat-util.h b/git-compat-util.h
index 5d543d2..7b4feae 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -138,11 +138,14 @@ extern unsigned char sane_ctype[256];
#define GIT_SPACE 0x01
#define GIT_DIGIT 0x02
#define GIT_ALPHA 0x04
+#define GIT_URIMARK 0x08
#define sane_istest(x,mask) ((sane_ctype[(unsigned char)(x)] & (mask)) != 0)
#define isspace(x) sane_istest(x,GIT_SPACE)
#define isdigit(x) sane_istest(x,GIT_DIGIT)
#define isalpha(x) sane_istest(x,GIT_ALPHA)
#define isalnum(x) sane_istest(x,GIT_ALPHA | GIT_DIGIT)
+/* Git extension - is character okay in URI (rfc2396, section 2.3), plus / */
+#define isuri(x) sane_istest(x,GIT_ALPHA | GIT_DIGIT | GIT_URIMARK)
#define tolower(x) sane_case((unsigned char)(x), 0x20)
#define toupper(x) sane_case((unsigned char)(x), 0)
^ permalink raw reply related
* Re: Broken PPC sha1.. (Re: Figured out how to get Mozilla into git)
From: Johannes Schindelin @ 2006-06-19 8:50 UTC (permalink / raw)
To: linux; +Cc: git, torvalds
In-Reply-To: <20060619084135.18632.qmail@science.horizon.com>
Hi,
On Mon, 19 Jun 2006, linux@horizon.com wrote:
> By the way, if anyone's still interested, I tried to produce a
> better-scheduled PowerPC sha1 core last year. Unfortunately, I don't have
> access to a PowerPC machine to test on, so debugging is a little painful.
If you have access to SourceForge's compile farm, they have a PPC-G5
there.
Ciao,
Dscho
^ permalink raw reply
* Re: Broken PPC sha1.. (Re: Figured out how to get Mozilla into git)
From: linux @ 2006-06-19 8:41 UTC (permalink / raw)
To: git, torvalds; +Cc: linux
By the way, if anyone's still interested, I tried to produce a
better-scheduled PowerPC sha1 core last year. Unfortunately, I don't have
access to a PowerPC machine to test on, so debugging is a little painful.
The latest version is appended, if anyone with a PowerPC machine wants to
try it out. It should drop in for ppc/sha1ppc.S. Hopefully the comments
explain the general idea.
Notes I should add to the comments:
- When reading the assembly code, note that PPC assembly permits a bare
number (no %r prefix) as a register number, and further that number can
be an expression! It makes the register renaming nice and simple.
- Also for folks unfamiliar, it's dest,src,src operand order.
- For a reminder, the PowerPC calling convention is:
%r0 - Temp. Always reads as zero in some contexts.
%r1 - stack pointer
%r2 - Confusing. Different documents say different things.
%r3..%r10 - Incoming arguments. Volatile across function calls.
%r11..%r12 - Have some special uses not relevant here. Volatile.
%r13..%r31 - Callee-save registers.
%lr, %ctr - Volatile. %lr holds return address on input.
And the way registers are used in this function are:
%r0 - Temp.
%r1 - Stack pointer. Used only for register saving.
%r2 - Not used.
%r3 - Points to hash accumulator A..E in memory.
%r4 - Points to data being hashed.
%r5 - Incoming loop count. Holds round constant K in body of loop.
%r6..%r10 - Working copies of A..E
%r11..%r26 - The W[] array of 16 input words being hashed
%r27..%r31 - Start-of-round copies of A..E.
%ctr - Holds loop count, copied from incoming %r5
%lr - Holds return address. Not modified.
- While I try to use the load/store multiple instructions where
appropriate, they have a severe penalty for unaligned operands
(they're microcoded optimistically, so do a full failing aligned
load before being re-issued as a slow-but-safe unaligned sequence),
and thanks to git's object type prefix, the source data is generally
unaligned, so they're deliberately NOT used to load the 16 words of
data hashed each iteration.
/*
* SHA-1 implementation for PowerPC.
*
* Copyright (C) 2005 Paul Mackerras <paulus@samba.org>
*/
/*
* We roll the registers for A, B, C, D, E around on each
* iteration; E on iteration t is D on iteration t+1, and so on.
* We use registers 6 - 10 for this. (Registers 27 - 31 hold
* the previous values.)
*/
#define RA(t) (((t)+4)%5+6)
#define RB(t) (((t)+3)%5+6)
#define RC(t) (((t)+2)%5+6)
#define RD(t) (((t)+1)%5+6)
#define RE(t) (((t)+0)%5+6)
/* We use registers 11 - 26 for the W values */
#define W(t) ((t)%16+11)
/* Register 5 is used for the constant k */
/*
* There are three F functions, used four groups of 20:
* - 20 rounds of f0(b,c,d) = "bit wise b ? c : d" = (^b & d) + (b & c)
* - 20 rounds of f1(b,c,d) = b^c^d = (b^d)^c
* - 20 rounds of f2(b,c,d) = majority(b,c,d) = (b&d) + ((b^d)&c)
* - 20 more rounds of f1(b,c,d)
*
* These are all scheduled for near-optimal performance on a G4.
* The G4 is a 3-issue out-of-order machine with 3 ALUs, but it can only
* *consider* starting the oldest 3 instructions per cycle. So to get
* maximum performace out of it, you have to treat it as an in-order
* machine. Which means interleaving the computation round t with the
* computation of W[t+4].
*
* The first 16 rounds use W values loaded directly from memory, while the
* remianing 64 use values computed from those first 16. We preload
* 4 values before starting, so there are three kinds of rounds:
* - The first 12 (all f0) also load the W values from memory.
* - The next 64 compute W(i+4) in parallel. 8*f0, 20*f1, 20*f2, 16*f1.
* - The last 4 (all f1) do not do anything with W.
*
* Therefore, we have 6 different round functions:
* STEPD0_LOAD(t,s) - Perform round t and load W(s). s < 16
* STEPD0_UPDATE(t,s) - Perform round t and compute W(s). s >= 16.
* STEPD1_UPDATE(t,s)
* STEPD2_UPDATE(t,s)
* STEPD1(t) - Perform round t with no load or update.
*
* The G5 is more fully out-of-order, and can find the parallelism
* by itself. The big limit is that it has a 2-cycle ALU latency, so
* even though it's 2-way, the code has to be scheduled as if it's
* 4-way, which can be a limit. To help it, we try to schedule the
* read of RA(t) as late as possible so it doesn't stall waiting for
* the previous round's RE(t-1), and we try to rotate RB(t) as early
* as possible while reading RC(t) (= RB(t-1)) as late as possible.
*/
/* the initial loads. */
#define LOADW(s) \
lwz W(s),(s)*4(%r4)
/*
* This is actually 13 instructions, which is an awkward fit,
* and uses W(s) as a temporary before loading it.
*/
#define STEPD0_LOAD(t,s) \
add RE(t),RE(t),W(t); andc %r0,RD(t),RB(t); /* spare slot */ \
add RE(t),RE(t),%r0; and W(s),RC(t),RB(t); rotlwi %r0,RA(t),5; \
add RE(t),RE(t),W(s); add %r0,%r0,%r5; rotlwi RB(t),RB(t),30; \
add RE(t),RE(t),%r0; lwz W(s),(s)*4(%r4);
/*
* This can execute starting with 2 out of 3 possible moduli, so it
* does 2 rounds in 9 cycles, 4.5 cycles/round.
*/
#define STEPD0_UPDATE(t,s) \
add RE(t),RE(t),W(t); andc %r0,RD(t),RB(t); xor W(s),W((s)-16),W((s)-3); \
add RE(t),RE(t),%r0; and %r0,RC(t),RB(t); xor W(s),W(s),W((s)-8); \
add RE(t),RE(t),%r0; rotlwi %r0,RA(t),5; xor W(s),W(s),W((s)-14); \
add RE(t),RE(t),%r5; rotlwi RB(t),RB(t),30; rotlwi W(s),W(s),1; \
add RE(t),RE(t),%r0;
/* Nicely optimal. Conveniently, also the most common. */
#define STEPD1_UPDATE(t,s) \
add RE(t),RE(t),W(t); xor %r0,RD(t),RB(t); xor W(s),W((s)-16),W((s)-3); \
add RE(t),RE(t),%r5; xor %r0,%r0,RC(t); xor W(s),W(s),W((s)-8); \
add RE(t),RE(t),%r0; rotlwi %r0,RA(t),5; xor W(s),W(s),W((s)-14); \
add RE(t),RE(t),%r0; rotlwi RB(t),RB(t),30; rotlwi W(s),W(s),1;
/*
* The naked version, no UPDATE, for the last 4 rounds. 3 cycles per.
* We could use W(s) as a temp register, but we don't need it.
*/
#define STEPD1(t) \
/* spare slot */ add RE(t),RE(t),W(t); xor %r0,RD(t),RB(t); \
rotlwi RB(t),RB(t),30; add RE(t),RE(t),%r5; xor %r0,%r0,RC(t); \
add RE(t),RE(t),%r0; rotlwi %r0,RA(t),5; /* idle */ \
add RE(t),RE(t),%r0;
/* 5 cycles per */
#define STEPD2_UPDATE(t,s) \
add RE(t),RE(t),W(t); and %r0,RD(t),RB(t); xor W(s),W((s)-16),W((s)-3); \
add RE(t),RE(t),%r0; xor %r0,RD(t),RB(t); xor W(s),W(s),W((s)-8); \
add RE(t),RE(t),%r5; and %r0,%r0,RC(t); xor W(s),W(s),W((s)-14); \
add RE(t),RE(t),%r0; rotlwi %r0,RA(t),5; rotlwi W(s),W(s),1; \
add RE(t),RE(t),%r0; rotlwi RB(t),RB(t),30;
#define STEP0_LOAD4(t,s) \
STEPD0_LOAD(t,s); \
STEPD0_LOAD((t+1),(s)+1); \
STEPD0_LOAD((t)+2,(s)+2); \
STEPD0_LOAD((t)+3,(s)+3);
#define STEPUP4(fn, t, s) \
STEP##fn##_UPDATE(t,s); \
STEP##fn##_UPDATE((t)+1,(s)+1); \
STEP##fn##_UPDATE((t)+2,(s)+2); \
STEP##fn##_UPDATE((t)+3,(s)+3); \
#define STEPUP20(fn, t, s) \
STEPUP4(fn, t, s); \
STEPUP4(fn, (t)+4, (s)+4); \
STEPUP4(fn, (t)+8, (s)+8); \
STEPUP4(fn, (t)+12, (s)+12); \
STEPUP4(fn, (t)+16, (s)+16)
.globl sha1_core
sha1_core:
stwu %r1,-80(%r1)
stmw %r13,4(%r1)
/* Load up A - E */
lmw %r27,0(%r3)
mtctr %r5
1:
lis %r5,0x5a82 /* K0-19 */
mr RA(0),%r27
LOADW(0)
mr RB(0),%r28
LOADW(1)
mr RC(0),%r29
LOADW(2)
ori %r5,%r5,0x7999
mr RD(0),%r30
LOADW(3)
mr RE(0),%r31
STEP0_LOAD4(0, 4)
STEP0_LOAD4(4, 8)
STEP0_LOAD4(8, 12)
STEPUP4(D0, 12, 16)
STEPUP4(D0, 16, 20)
lis %r5,0x6ed9 /* K20-39 */
ori %r5,%r5,0xeba1
STEPUP20(D1, 20, 24)
lis %r5,0x8f1b /* K40-59 */
ori %r5,%r5,0xbcdc
STEPUP20(D2, 40, 44)
lis %r5,0xca62 /* K60-79 */
ori %r5,%r5,0xc1d6
STEPUP4(D1, 60, 64)
STEPUP4(D1, 64, 68)
STEPUP4(D1, 68, 72)
STEPUP4(D1, 72, 76)
STEPD1(76)
STEPD1(77)
STEPD1(78)
STEPD1(79)
/* Add results to original values */
add %r31,%r31,RE(0)
add %r30,%r30,RD(0)
add %r29,%r29,RC(0)
add %r28,%r28,RB(0)
add %r27,%r27,RA(0)
addi %r4,%r4,64
bdnz 1b
/* Save final hash, restore registers, and return */
stmw %r27,0(%r3)
lmw %r13,4(%r1)
addi %r1,%r1,80
blr
^ permalink raw reply
* Re: What's in git.git
From: Johannes Schindelin @ 2006-06-19 8:35 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Petr Baudis, git
In-Reply-To: <7vy7vta9ae.fsf@assigned-by-dhcp.cox.net>
Hi,
On Mon, 19 Jun 2006, Junio C Hamano wrote:
> Petr Baudis <pasky@suse.cz> writes:
>
> > Dear diary, on Sun, Jun 18, 2006 at 02:26:14PM CEST, I got a letter
> > where Johannes Schindelin <Johannes.Schindelin@gmx.de> said that...
> >
> >> Note that this issue is orthogonal to the need for a user-specific config
> >> file. I still think that this one should go in.
> >
> > I agree as well.
>
> OK, let's have it then.
>
> Johannes, this makes your earlier ~/.gitconfig and --no-local
> patches no longer applicable, so I'd drop them from "pu" for
> now. But I suspect we do want to have per user configuration
> that is used as a fallback for per repo configuration, right?
I'll redo the patches, then.
Ciao,
Dscho
^ permalink raw reply
* Re: simple use case scenario for --read-tree and --write-tree with --prefix option
From: Aneesh Kumar @ 2006-06-19 7:58 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vpsh5a8gs.fsf@assigned-by-dhcp.cox.net>
On 6/19/06, Junio C Hamano <junkio@cox.net> wrote:
> "Aneesh Kumar" <aneesh.kumar@gmail.com> writes:
>
> > I searched the archives but didn't find anything. If i understand
> > correctly the sub project idea is using the gitlink object type. So
> > what is read-tree and write-tree with --prefix option supposed to
> > achieve.
>
> The --prefix option to read-tree was very useful when I did a
> hand merge of gitweb for example. I am reasonably sure clever
> people will find other uses as well.
>
Ok i tried this
mkdir test/test1
mkdir test/test2
cd test
git init-db
cd test1
git init-db
cd ../test2
git init-db
now i do some devel under test2
now if i want to pull this with the history to toplevel directory test
should i be doing
cd test
kvaneesh@satan:/tmp/test$ git read-tree --prefix=test1/ $(cat
test1/.git/refs/heads/master)
fatal: failed to unpack tree object c6c049d03f0bee0ac546ff6e436d5f6f3a5f4864
But the above command doesn't work for me. I guess i am missing something.
-aneesh
^ permalink raw reply
* Re: simple use case scenario for --read-tree and --write-tree with --prefix option
From: Junio C Hamano @ 2006-06-19 7:52 UTC (permalink / raw)
To: Aneesh Kumar; +Cc: git
In-Reply-To: <cc723f590606190028t65c76c74t6a90d1dcec411598@mail.gmail.com>
"Aneesh Kumar" <aneesh.kumar@gmail.com> writes:
> I searched the archives but didn't find anything. If i understand
> correctly the sub project idea is using the gitlink object type. So
> what is read-tree and write-tree with --prefix option supposed to
> achieve.
The --prefix option to read-tree was very useful when I did a
hand merge of gitweb for example. I am reasonably sure clever
people will find other uses as well.
^ permalink raw reply
* Re: What's in git.git
From: Junio C Hamano @ 2006-06-19 7:34 UTC (permalink / raw)
To: Petr Baudis; +Cc: Johannes Schindelin, git
In-Reply-To: <20060618130837.GN2609@pasky.or.cz>
Petr Baudis <pasky@suse.cz> writes:
> Dear diary, on Sun, Jun 18, 2006 at 02:26:14PM CEST, I got a letter
> where Johannes Schindelin <Johannes.Schindelin@gmx.de> said that...
>
>> Note that this issue is orthogonal to the need for a user-specific config
>> file. I still think that this one should go in.
>
> I agree as well.
OK, let's have it then.
Johannes, this makes your earlier ~/.gitconfig and --no-local
patches no longer applicable, so I'd drop them from "pu" for
now. But I suspect we do want to have per user configuration
that is used as a fallback for per repo configuration, right?
I think we should mark what we have in "next" tonight, plus a
few others, as v1.4.1 sometime this coming week and continue
from there.
^ permalink raw reply
* simple use case scenario for --read-tree and --write-tree with --prefix option
From: Aneesh Kumar @ 2006-06-19 7:28 UTC (permalink / raw)
To: Git Mailing List
Hi all,
I searched the archives but didn't find anything. If i understand
correctly the sub project idea is using the gitlink object type. So
what is read-tree and write-tree with --prefix option supposed to
achieve.
-aneesh
^ permalink raw reply
* Re: [PATCH] Fix PPC SHA1 routine for large input buffers
From: Linus Torvalds @ 2006-06-19 5:02 UTC (permalink / raw)
To: Paul Mackerras; +Cc: Martin Langhoff, Nicolas Pitre, Jon Smirl, git
In-Reply-To: <17557.57564.267469.561683@cargo.ozlabs.ibm.com>
On Mon, 19 Jun 2006, Paul Mackerras wrote:
>
> The PPC SHA1 routine had an overflow which meant that it gave
> incorrect results for input buffers >= 512MB. This fixes it by
> ensuring that the update of the total length in bits is done using
> 64-bit arithmetic.
>
> Signed-off-by: Paul Mackerras <paulus@samba.org>
Acked-by: Linus Torvalds <torvalds@osdl.org>
This fixes git-fsck-objects for me on the mozilla archive, no more
complaints about bad SHA1's.
And yeah, now it's taking me 14 minutes too, so the 7-minute fsck was just
because it didn't actually check the SHA1 of the large pack fully.
(Which is actually good news - half of the time is literally checking the
pack integrity. That implies that the individual object integrity isn't as
dominating as I thought it would be, and that things like hw-accelerated
SHA1 engines will help with fsck. I'd not be surprised to see things like
that in a couple of years).
Linus
^ permalink raw reply
* Re: git 1.4.0 usability problem
From: Junio C Hamano @ 2006-06-19 3:11 UTC (permalink / raw)
To: Jeff Garzik; +Cc: git
In-Reply-To: <4495DB3B.10403@garzik.org>
Jeff Garzik <jeff@garzik.org> writes:
> But if what Ryan says is true, about simply needing to ditch
> the "-f" argument I habitually pass to 'git checkout', would that
> alleviate the need for a patch?
To a certain degree, yes.
But I suspect (I am not a kernel person so I can only speculate)
in the kernel workflow you would often pick up a patch from the
list, apply it to your working tree (without applying it to your
index, IOW with "patch -p1" or "git apply", not with "git apply
--index"), and then decide to pull from somewhere else while
your working tree is dirty (but index is not). The patch might
have created a new file or two, and the pull may also contain a
commit that applied the same patch in question. The no-clobber
check would trigger in such a case preventing you from pulling,
and neither "checkout" nor "checkout -f" would clean these new
files that you have not told git about.
> FWIW, my workflow is
>
> cd /repos
> cd linux-2.6
> git pull
> cd ../libata-dev
> git checkout -f master # guarantee any WIP goes away
We kept saying "with checkout -f any dirty state goes away from
your working tree". It is true only with respect to the files
git knows about. The trouble you experienced was about
untracked files -- files git does not know about, and they will
be left behind.
So if path F is in test branch head and linus branch head, but
not in your master branch head, and you have checked out test in
your working tree, even if path F in the working tree is clean:
git checkout -f master
will leave F behind. If you pull from linus at this point, the
check would trigger. Running "git checkout master" without -f
would however remove it and you would not have the problem.
That is what Ryan's suggestion is about.
However, if you have a patch you got from somebody on the net
that creates F (maybe it is the same patch linus accepted
recently) while on your master branch, and you tried to examine
it by applying it to your working tree with "patch -p1" or "git
apply", your working tree will have F that is not in index (and
in your branch head). In that state if you pull from linus, the
no-clobber check triggers. In this case, neither "git checkout
master", "git checkout -f master", nor "git reset --hard master"
would remove F, because git does not even know about it, so
pulling from linus would fail. "git clean" would removes F, so
it may not be a big deal, but it is rather a heavy-handed
operation that removes all crufts, so you may find it a
not-so-useful workaround (I certainly would, and that is the
primary reason I rarely use "git clean" myself).
It's a bit sad situation. One of the useful feature of git is
that you can continue working in a dirty working tree as long as
your index is clean and your local changes do not interfere with
a merge, patch application, or branch switching. Strictly
speaking, this no-clobber check _is_ about a local change that
does interfere with the operation, so from theoretical point of
view it is a good safety measure, but at the same time we did
not consider untracked files as precious until recently, and I
suspect that "the same patch applied elsewhere to create the
same file" pattern is reasonably common that this safety valve
may interfere the work more often than it may help avoiding
mistakes.
^ permalink raw reply
* [PATCH] Fix PPC SHA1 routine for large input buffers
From: Paul Mackerras @ 2006-06-18 23:25 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Martin Langhoff, Nicolas Pitre, Jon Smirl, git
In-Reply-To: <Pine.LNX.4.64.0606181543270.5498@g5.osdl.org>
The PPC SHA1 routine had an overflow which meant that it gave
incorrect results for input buffers >= 512MB. This fixes it by
ensuring that the update of the total length in bits is done using
64-bit arithmetic.
Signed-off-by: Paul Mackerras <paulus@samba.org>
---
Linus Torvalds writes:
> Paul - I think the ppc SHA1_Update() overflows in 32 bits, when the length
> of the memory area to be checksummed is huge.
Yep. I checked the assembly output of this, and it looks right, but I
haven't actually tested it by running it...
Paul.
diff --git a/ppc/sha1.c b/ppc/sha1.c
index 5ba4fc5..0820398 100644
--- a/ppc/sha1.c
+++ b/ppc/sha1.c
@@ -30,7 +30,7 @@ int SHA1_Update(SHA_CTX *c, const void *
unsigned long nb;
const unsigned char *p = ptr;
- c->len += n << 3;
+ c->len += (uint64_t) n << 3;
while (n != 0) {
if (c->cnt || n < 64) {
nb = 64 - c->cnt;
^ permalink raw reply related
* Re: git 1.4.0 usability problem
From: Jeff Garzik @ 2006-06-18 23:01 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, Ryan Anderson
In-Reply-To: <7vbqsqdru0.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano wrote:
> Jeff Garzik <jeff@garzik.org> writes:
>
>> Here is how to reproduce:
>
> This is not related to the "not clobbering untracked files"
> safety valve under discussion, but one thing I noticed.
>
>> git clone -l $url/torvalds/linux-2.6.git tmp-2.6
>> cd tmp-2.6
>> cp .git/refs/tags/v2.6.12 .git/refs/heads/tmp
>> git checkout -f tmp
>
> This should never have been supported. At this point tmp is a
> tag object that is under heads/ -- a definite no-no. We should
> make checkout more careful to complain about it.
>
> Doing
>
> git update-ref refs/heads/tmp $(git rev-parse v2.6.12^0)
>
> instead of "cp" is kosher, and
>
> git-rev-parse v2.6.12^0 >.git/refs/heads/tmp
>
> is OK under the current implementation of refs.
Sorry about that. The contrived example produced the same results as
the real-world example (updating jgarzik/{libata-dev,scsilun-2.6}.git
branches).
>> git pull . master
>> # watch OBVIOUS FAST-FORWARD MERGE complain about untracked
>> # working tree files
>
> In any case, here is a patch I think would alleviate your
> original problem.
>
> Sorry for the trouble. I really did not want to disrupt the
> workflow of old timers in the name of making it safer for new
> people. Could you comment on whether this is an acceptable
> approach?
>
> -- >8 --
> [PATCH] Conditionally loosen "no clobber untracked files" safety valve.
>
> This introduces a new configuration item "core.oktoclobber" to
> control how untracked working tree file is handled during branch
> switching.
>
> The safety valve introduced during 1.4.0 development cycle
> refrains from checking out a file that exists in the working
> tree, not in the current HEAD tree and exists in the branch we
> are switching to, in order to prevent accidental and
> irreversible lossage of user data. This can be controlled by
> having core.oktoclobber configuration item:
I'm a bit under the weather today, so I must defer thinking about this.
:) But if what Ryan says is true, about simply needing to ditch the
"-f" argument I habitually pass to 'git checkout', would that alleviate
the need for a patch?
FWIW, my workflow is
cd /repos
cd linux-2.6
git pull
cd ../libata-dev
git checkout -f master # guarantee any WIP goes away
git pull ../linux-2.6 # update vanilla branch
git checkout -f upstream# switch to working branch,
# guarantee any WIP goes away.
git pull . master # pull latest upstream updates
build/test/etc.
git checkout -f sii-m15w # switch to topic-specific branch,
# whose parent is always #upstream
git pull . upstream
build/test/etc.
repeat for several topics (on-going devel branches)
git checkout -f -b ALL upstream # create everything-together
# test branch
git pull . sii-m15w
git pull . topicB
git pull . topicC
build/test/etc.
git checkout -f master
./push # calls 'git push --force --all $url'
More tomorrow,
Jeff
^ permalink raw reply
* Broken PPC sha1.. (Re: Figured out how to get Mozilla into git)
From: Linus Torvalds @ 2006-06-18 22:51 UTC (permalink / raw)
To: Martin Langhoff, Paul Mackerras; +Cc: Nicolas Pitre, Jon Smirl, git
In-Reply-To: <Pine.LNX.4.64.0606181532130.5498@g5.osdl.org>
On Sun, 18 Jun 2006, Linus Torvalds wrote:
>
> On Mon, 19 Jun 2006, Martin Langhoff wrote:
> >
> > No problems here with my latest import run. fsck-objects --full comes
> > clean, takes 14m:
> >
> > /usr/bin/time git-fsck-objects --full
> > 737.22user 38.79system 14:09.40elapsed 91%CPU (0avgtext+0avgdata 0maxresident)k
> > 0inputs+0outputs (20807major+19483471minor)pagefaults 0swaps
>
> It takes much less than that for me:
>
> 408.40user 32.56system 7:22.07elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
> 0inputs+0outputs (145major+13455672minor)pagefaults 0swaps
Ok, re-building the thing with MOZILLA_SHA1=1 rather than my default
PPC_SHA1=1 fixes the problem. I no longer get that "SHA1 mismatch with
itself" on the pack-file.
Sadly, it also takes a _lot_ longer to fsck.
Paul - I think the ppc SHA1_Update() overflows in 32 bits, when the length
of the memory area to be checksummed is huge.
In particular, the pack-file is 535MB in size, and the way we check the
SHA1 checksum is by just mapping it all, doing a single SHA1_Update() over
the whole pack-file, and comparing the end result with the internal SHA1
at the end of the pack-file.
The PPC SHA1_Update() function starts off with:
int SHA1_Update(SHA_CTX *c, const void *ptr, unsigned long n)
{
...
c->len += n << 3;
which will obviously overflow if "n" is bigger than 29 bits, ie 512MB.
So doing the length in bits (or whatever that "<<3" is there for) doesn't
seem to be such a great idea.
I guess we could make the caller just always chunk it up, but wouldn't it
be nice to fix the PPC SHA1 implementation instead?
That said, the _only_ thing this will ever trigger on in practice is
exactly this one case: a large packfile whose checksum was _correctly_
generated - because pack-file generation does it in IO chunks using the
csum-file interfaces - but that will be incorrectly checked because we
check it all at once.
So as bugs go, it's a fairly benign one.
Linus
^ permalink raw reply
* Re: Figured out how to get Mozilla into git
From: Linus Torvalds @ 2006-06-18 22:36 UTC (permalink / raw)
To: Martin Langhoff; +Cc: Nicolas Pitre, Jon Smirl, git
In-Reply-To: <46a038f90606181440q4fd03bebl9495ace131eb958@mail.gmail.com>
On Mon, 19 Jun 2006, Martin Langhoff wrote:
>
> No problems here with my latest import run. fsck-objects --full comes
> clean, takes 14m:
>
> /usr/bin/time git-fsck-objects --full
> 737.22user 38.79system 14:09.40elapsed 91%CPU (0avgtext+0avgdata 0maxresident)k
> 0inputs+0outputs (20807major+19483471minor)pagefaults 0swaps
It takes much less than that for me:
408.40user 32.56system 7:22.07elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (145major+13455672minor)pagefaults 0swaps
and in particular note the much lower minor pagefaults number (which is a
very good approximation of total RSS). Mine is with all the memory
optimizations in place, but I didn't see _that_ big of a difference, so
there's something else in addition.
However, the fact that I get "SHA1 mismatch with itself" is strange. The
re-pack will always re-generate the SHA1, so I worry that this is perhaps
some PPC-specific bug in SHA1 handling (and it's entirely possible that
it's triggered by doing a SHA1 over a 500+MB area).
The fact that you don't see it is indicative that it's somehow specific to
my setup.
> BTW, that import (with the latest code Junio has) took 37hs even with
> the aggressive repack -a -d. I want to bench it dropping the -a from
> the recurrring repack, and doing a final repack -a -d.
Yeah, that's probably the right thing to do. The "-a" is ok with tons of
memory, and I'm trying to make it ok with _less_ memory, but it's probably
just not worth it.
Linus
^ permalink raw reply
* Re: git 1.4.0 usability problem
From: Ryan Anderson @ 2006-06-18 22:27 UTC (permalink / raw)
To: Ryan Anderson; +Cc: Jeff Garzik, Git Mailing List
In-Reply-To: <20060618164300.GI25520@h4x0r5.com>
On Sun, Jun 18, 2006 at 09:43:00AM -0700, Ryan Anderson wrote:
>
> The fix is to drop the "-f" from git checkout, and things should work
> correctly. ("-f" should really not be a normal thing to use. For
> switching branches, "git checkout" should be sufficient, and should
> result ina working tree that doesn't contain nearly as many potential
> conflict sources.
I wrote all of that from memory, but I figured I should really test it:
$ git branch tmp v2.6.12
$ git checkout tmp
$ git pull . master
Updating from 9ee1c939d1cb936b1f98e8d81aeffab57bae46ab to
553698f944ed715dfe023b4cef07601f0ce735f0
Checking files out...
100% (14284/14284) done
Fast forward
(Watch insanely large diffstat blow by)
$ git status
# On branch refs/heads/tmp
nothing to commit
$ git branch
master
origin
ryan
* tmp
So, I think all you need to do is drop the "-f" from the call to
checkout, and the issues will be fixed, for your particular use case.
--
Ryan Anderson
sometimes Pug Majere
^ permalink raw reply
* Re: git 1.4.0 usability problem
From: Junio C Hamano @ 2006-06-18 22:25 UTC (permalink / raw)
To: Jeff Garzik; +Cc: git, Ryan Anderson
In-Reply-To: <449557B6.1080907@garzik.org>
Jeff Garzik <jeff@garzik.org> writes:
> Here is how to reproduce:
This is not related to the "not clobbering untracked files"
safety valve under discussion, but one thing I noticed.
> git clone -l $url/torvalds/linux-2.6.git tmp-2.6
> cd tmp-2.6
> cp .git/refs/tags/v2.6.12 .git/refs/heads/tmp
> git checkout -f tmp
This should never have been supported. At this point tmp is a
tag object that is under heads/ -- a definite no-no. We should
make checkout more careful to complain about it.
Doing
git update-ref refs/heads/tmp $(git rev-parse v2.6.12^0)
instead of "cp" is kosher, and
git-rev-parse v2.6.12^0 >.git/refs/heads/tmp
is OK under the current implementation of refs.
> git pull . master
> # watch OBVIOUS FAST-FORWARD MERGE complain about untracked
> # working tree files
In any case, here is a patch I think would alleviate your
original problem.
Sorry for the trouble. I really did not want to disrupt the
workflow of old timers in the name of making it safer for new
people. Could you comment on whether this is an acceptable
approach?
-- >8 --
[PATCH] Conditionally loosen "no clobber untracked files" safety valve.
This introduces a new configuration item "core.oktoclobber" to
control how untracked working tree file is handled during branch
switching.
The safety valve introduced during 1.4.0 development cycle
refrains from checking out a file that exists in the working
tree, not in the current HEAD tree and exists in the branch we
are switching to, in order to prevent accidental and
irreversible lossage of user data. This can be controlled by
having core.oktoclobber configuration item:
- When core.oktoclobber is set to "false" (the default),
untracked working tree files are never overwritten.
- When core.oktoclobber is set to "true", the check is
disabled.
- When core.oktoclobber is set to "ask", and both standard
input and standard error streams are connected to the
terminal, we ask the user if it is OK to clobber. You can
answer:
y: to allow clobbering only one path; the question is
asked again for other paths.
n: to stop the operation.
a: to allow clobbering any untracked files; the question
is not asked again.
If the configuration item is set to "ask" but the program is
not talking to a terminal, it refrains from clobbering the
untracked files.
Signed-off-by: Junio C Hamano <junkio@cox.net>
---
builtin-read-tree.c | 43 ++++++++++++++++++++++++++++++++++++++++++-
cache.h | 6 ++++++
config.c | 20 ++++++++++++++++++++
environment.c | 1 +
4 files changed, 69 insertions(+), 1 deletions(-)
diff --git a/builtin-read-tree.c b/builtin-read-tree.c
index 04506da..7a7018c 100644
--- a/builtin-read-tree.c
+++ b/builtin-read-tree.c
@@ -477,6 +477,46 @@ static void invalidate_ce_path(struct ca
cache_tree_invalidate_path(active_cache_tree, ce->name);
}
+static int ok_to_clobber_untracked(const char *path, const char *action)
+{
+ switch (ok_to_clobber_untracked_files) {
+ case NEVER_CLOBBER:
+ return 0;
+ case OK_TO_CLOBBER:
+ return 1;
+ case ASK_TO_CLOBBER:
+ if (isatty(0) && isatty(2)) {
+ char answer[1024];
+ while (1) {
+ answer[0] = '\0';
+ fprintf(stderr,
+ "Untracked working tree file '%s' is"
+ " about to be %s. Is it OK "
+ "[y]es/[n]o/yes to [a]ll? ",
+ path, action);
+ fgets(answer, sizeof(answer), stdin);
+ switch (answer[0]) {
+ case 'y': case 'Y':
+ return 1;
+ case 'n': case 'N':
+ return 0;
+ case 'a': case 'A':
+ ok_to_clobber_untracked_files =
+ OK_TO_CLOBBER;
+ return 1;
+ default:
+ fprintf(stderr,
+ "I do not understand.\n");
+ }
+ }
+ }
+ else {
+ ok_to_clobber_untracked_files = NEVER_CLOBBER;
+ return 0;
+ }
+ }
+}
+
/*
* We do not want to remove or overwrite a working tree file that
* is not tracked.
@@ -485,7 +525,8 @@ static void verify_absent(const char *pa
{
struct stat st;
- if (index_only || reset || !update)
+ if (index_only || reset || !update ||
+ ok_to_clobber_untracked(path, action))
return;
if (!lstat(path, &st))
die("Untracked working tree file '%s' "
diff --git a/cache.h b/cache.h
index f630cf4..7468440 100644
--- a/cache.h
+++ b/cache.h
@@ -183,6 +183,12 @@ extern int log_all_ref_updates;
extern int warn_ambiguous_refs;
extern int diff_rename_limit_default;
extern int shared_repository;
+enum ok_to_clobber {
+ NEVER_CLOBBER = 0,
+ OK_TO_CLOBBER,
+ ASK_TO_CLOBBER
+};
+extern enum ok_to_clobber ok_to_clobber_untracked_files;
extern const char *apply_default_whitespace;
#define GIT_REPO_VERSION 0
diff --git a/config.c b/config.c
index 984c75f..13f5f4f 100644
--- a/config.c
+++ b/config.c
@@ -251,6 +251,21 @@ int git_config_bool(const char *name, co
return git_config_int(name, value) != 0;
}
+static enum ok_to_clobber git_config_clobber(const char *var, const char *value)
+{
+ if (!strcasecmp(value, "ask"))
+ return ASK_TO_CLOBBER;
+ if (!strcasecmp(value, "yes"))
+ return OK_TO_CLOBBER;
+ if (!strcasecmp(value, "ok"))
+ return NEVER_CLOBBER;
+ if (!strcasecmp(value, "no"))
+ return NEVER_CLOBBER;
+ if (git_config_bool(var, value))
+ return OK_TO_CLOBBER;
+ return NEVER_CLOBBER;
+}
+
int git_default_config(const char *var, const char *value)
{
/* This needs a better name */
@@ -279,6 +294,11 @@ int git_default_config(const char *var,
return 0;
}
+ if (!strcmp(var, "core.oktoclobber")) {
+ ok_to_clobber_untracked_files = git_config_clobber(var, value);
+ return 0;
+ }
+
if (!strcmp(var, "user.name")) {
safe_strncpy(git_default_name, value, sizeof(git_default_name));
return 0;
diff --git a/environment.c b/environment.c
index 2e79eab..c388b5b 100644
--- a/environment.c
+++ b/environment.c
@@ -19,6 +19,7 @@ int warn_ambiguous_refs = 1;
int repository_format_version = 0;
char git_commit_encoding[MAX_ENCODING_LENGTH] = "utf-8";
int shared_repository = 0;
+extern enum ok_to_clobber ok_to_clobber_untracked_files = NEVER_CLOBBER;
const char *apply_default_whitespace = NULL;
static char *git_dir, *git_object_dir, *git_index_file, *git_refs_dir,
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox