* Re: full kernel history, in patchset format
From: Ingo Molnar @ 2005-04-16 19:41 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <Pine.LNX.4.58.0504160953310.7211@ppc970.osdl.org>
* Linus Torvalds <torvalds@osdl.org> wrote:
> > the history data starts at 2.4.0 and ends at 2.6.12-rc2. I've included a
> > script that will apply all the patches in order and will create a
> > pristine 2.6.12-rc2 tree.
>
> Hey, that's great. I got the CVS repo too, and I was looking at it,
> but the more I looked at it, the more I felt that the main reason I
> want to import it into git ends up being to validate that my size
> estimates are at all realistic.
>
> I see that Thomas Gleixner seems to have done that already, and come
> to a figure of 3.2GB for the last three years, which I'm very happy
> with, mainly because it seems to match my estimates to a tee. [...]
(yeah, we apparently worked in parallel - i only learned about his
efforts after i sent my mail. He was using BK to extract info, i was
using the CVS tree alone and no BK code whatsoever. (I dont think there
will be any argument about who owns what, but i wanted to be on the safe
side, and i also wanted to see how complete and usable the CVS metadata
is - it's close to perfect i'd say, for the purposes i care about.))
> But I wonder if we actually want to actually populate the whole
> history..
yeah, it definitely feels a bit brave to import 28,000 changesets into a
source-code database project that will be a whopping 2 weeks old in 2
days ;) Even if we felt 100% confident about all the basics (which we do
of course ;), it's just simply too young to tie things down via a 3.2GB
database. It feels much more natural to grow it gradually, 28,000
changesets i'm afraid would just suffocate the 'project growth
dynamics'. Not going too fast is just as important as not going too
slow.
I didnt generate the patchset to get it added into some central
repository right now, i generated it to check that we _do_ have all the
revision history in an easy to understand format which does generate
today's kernel tree, so that we can lean back and worry about the full
database once things get a bit more settled down (in a couple of months
or so). It's also an easy testbed for GIT itself.
but the revision history was one of the main reasons i used BK myself,
so we'll need a merged database eventually. Occasionally i needed to
check who was the one who touched a particular piece of code - was that
fantastic new line of code written by me, or was that buggy piece of
crap written by someone else? ;) Also, looking at a change and then
going to the changeset that did it, and then looking at the full picture
was pretty useful too. So that sort of annotation, and generally
navigating around _quickly_ and looking at the 'flow' of changes going
into a particular file was really useful (for me).
Ingo
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Mike Taht @ 2005-04-16 19:42 UTC (permalink / raw)
To: Junio C Hamano, git
In-Reply-To: <7v8y3i7cn9.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano wrote:
>>>>>>"MT" == Mike Taht <mike.taht@timesys.com> writes:
>
>
> MT> alternatively, "git-archive-torrent" to create a list of files for a
> MT> bittorrent feed....
>
> That is certainly good for establishing the baseline, but you
> still need to leverage the inherent delta-compressibility
> between related blobs/trees by also doing something like what I
> described as "diff package", don't you?
Yes... yes you could have files and diffs generated statically...
although something like a bittorrent server/client/frontend, call it
"gittorrent" (I hate being the first to make this pun) could walk the
hashes dynamically (
Ihave: sha,sha,sha,sha... Sendme: shaxxxxxxxxxxxxxxxxxxx
Hereswhatyouneedfromgit: file,file,file,diff,diff,diff,...)
--
Mike Taht
"It looks like blind screaming hedonism won out."
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Junio C Hamano @ 2005-04-16 19:44 UTC (permalink / raw)
To: Christopher Li; +Cc: Mike Taht, git
In-Reply-To: <20050416162632.GA3309@64m.dyndns.org>
>>>>> "CL" == Christopher Li <git@chrisli.org> writes:
CL> I bet 90% of the time people sync to the repository head first
CL> want to check out the last bits. And maybe reading some change
CL> log to see what is changed.
CL> So having all the commit object, the user will able to see
CL> what is change and which version he we like to check out.
Makes sense.
^ permalink raw reply
* Re: full kernel history, in patchset format
From: Daniel Barkalow @ 2005-04-16 20:19 UTC (permalink / raw)
To: Mike Taht; +Cc: Junio C Hamano, git
In-Reply-To: <42616A9F.1030302@timesys.com>
On Sat, 16 Apr 2005, Mike Taht wrote:
> Junio C Hamano wrote:
> >>>>>>"MT" == Mike Taht <mike.taht@timesys.com> writes:
> >
> >
> > MT> alternatively, "git-archive-torrent" to create a list of files for a
> > MT> bittorrent feed....
> >
> > That is certainly good for establishing the baseline, but you
> > still need to leverage the inherent delta-compressibility
> > between related blobs/trees by also doing something like what I
> > described as "diff package", don't you?
>
> Yes... yes you could have files and diffs generated statically...
>
> although something like a bittorrent server/client/frontend, call it
> "gittorrent" (I hate being the first to make this pun) could walk the
> hashes dynamically (
> Ihave: sha,sha,sha,sha... Sendme: shaxxxxxxxxxxxxxxxxxxx
> Hereswhatyouneedfromgit: file,file,file,diff,diff,diff,...)
I'm actually working on a trivial HTTP client to do this. The user says
"get <commit-id> from <url>", and it gets that object, the associated
trees, and the associated blobs, skipping any that it already has.
This should save having a non-standard public-facing server process, and
be essentially as effective, at least once I have it using a single
connection for everything.
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: Merge with git-pasky II.
From: Sanjoy Mahajan @ 2005-04-16 20:29 UTC (permalink / raw)
To: Linus Torvalds; +Cc: David Woodhouse, Junio C Hamano, Petr Baudis, git
In-Reply-To: <Pine.LNX.4.58.0504150753440.7211@ppc970.osdl.org>
> And that "where did this come from" decision should be done at _search_
> time, not commit time.
I like this elegant approach, but clever pattern matching can help even
at commit time. Suppose hello.c is simply:
printf ("Hello %d\n", year);
And then developer A updates hello.c to:
printf ("Hello %d\n", year);
printf ("And %d\n", year+1);
Meanwhile developer B updates hello.c to:
printf ("Hello %d\n", yyyy);
How to merge these two changes? The psychic solution is
printf ("Hello %d\n", yyyy);
printf ("And %d\n", yyyy+1);
Darcs handles token renames specially, but it's not a general solution
so let's leave it aside. The example does not have enough information
to make the psychic solution unique or reliable, but imagine that the
example were longer to solve that problem. You'd want to describe the
delta A(hello.c) as
1. duplicated message line
2. changed 2nd line a bit
And B(hello.c) as
1. Changed year to yyyy
In that representation, merging the two deltas becomes
1. duplicated message line
2. changed 2nd line a bit
3. Changed year to yyyy in both lines
Or, by commuting the merge operations and adjusting for their
non-commutativity (in terminology like darcs's -- I'm also a physicist):
1. Changed year to yyyy
2. duplicated message line
3. changed 2nd line a bit
So here some of the computation that Linus wants only at question time
(e.g. 'how did that line get here??') is also useful at merge time.
It's difficult (expensive, unreliable) to describe deltas in the form
above or, worse, to merge two such descriptions, but I hope it
illustrates the point. And perhaps a robust and easier-to-compute
change-description language can be dreamt up, even if the general
problem of describing changes compactly is not computable -- it's almost
the same, or is the same, as finding the Kolmogorov complexity of a data
set.
Or have I missed a fundamental point?
-Sanjoy
`A society of sheep must in time beget a government of wolves.'
- Bertrand de Jouvenal
^ permalink raw reply
* Re: Merge with git-pasky II.
From: Linus Torvalds @ 2005-04-16 20:41 UTC (permalink / raw)
To: Sanjoy Mahajan; +Cc: David Woodhouse, Junio C Hamano, Petr Baudis, git
In-Reply-To: <E1DMtuY-0002fL-3b@approximate.corpus.cam.ac.uk>
On Sat, 16 Apr 2005, Sanjoy Mahajan wrote:
>
> I like this elegant approach, but clever pattern matching can help even
> at commit time. Suppose hello.c is simply:
Here, what you're talking about is not "commit", but "merge".
The git model very much separates the two events. You first generate a
merged tree, an dyou commit that merge as a separate and largely totally
independent phase.
And yes, I agree that with merging, you do end up potentially wanting to
try different things. When I've done my "git" merges, all I've really done
is to make sure that the trivial parts basically merge in zero time, so
that you can afford to perhaps spend some effort on handling the _real_
merge conflicts.
Many systems seem to be designed around a "clever merge" algorithm, with
darcs perhaps being the most extreme example. The problem with that design
is that 99.9% of all the work is not at all about being clever, and if you
try to base your design around the clever things, your performance will
definitely suck.
So I think that with git, you can actually really try to be clever,
because when you get a merge conflict, you're now only worrying about one
file out of 17,000, and then you can go wild on that one and try different
merge algorithms (token merge, character-merge, line-based merge, you name
it).
Of course, I might not actually personally want to depend on any clever
merges, but the git infrastructure really doesn't care. My plumbing
doesn't merge the conflicts that arise within one single object, or the
filename differences - you can do anything you want on that.
Linus
^ permalink raw reply
* Re: SHA1 hash safety
From: Brian O'Mahoney @ 2005-04-16 21:35 UTC (permalink / raw)
To: C. Scott Ananian; +Cc: omb, David Lang, Ingo Molnar, git
In-Reply-To: <Pine.LNX.4.61.0504161040310.29343@cag.csail.mit.edu>
Please see below:
C. Scott Ananian wrote:
> On Sat, 16 Apr 2005, Brian O'Mahoney wrote:
>
>> (1) I _have_ seen real-life collisions with MD5, in the context of
>> Document management systems containing ~10^6 ms-WORD documents.
>
>
> Dude! You could have been *famous*! Why the
> aitch-ee-double-hockey-sticks didn't you publish this when you found it?
> Seriously, man.
The MD5 has was fine, or at least the code (a) produced the correct
results on the published test cases, and, (b) was properly applied to
all bytes of the file(s). I was surprised when it happened, which is why
I bothered to post to this list at this time, so I make two more points
(1) These hashes were designed, to assist in the construction of digital
signatures, ie so it is hard to produce a document to hash to a known
hash value, and that with a defined document format so they are designed
(i) hash similar documents far apart, and (ii) be hard to reverse;
it says nothing about naturally ocurring collisions, ie where the
document is not constrained to be similar,
>
> Even given the known weaknesses in MD5, it would take much more than a
> million documents to find MD5 collisions. I can only conclude that the
> hash was being used incorrectly; most likely truncated (my wild-ass
> guess would be to 32 bits; a collision is likely with > 50% probability
> in a million document store for a hash of less than 40 bits).
>
> I know the current state of the art here. It's going to take more than
> just hearsay to convince me that full 128-bit MD5 collisions are likely.
> I believe there are only two or so known to exist so far, and those were
> found by a research team in China (which, yes, is fairly famous among
> the cryptographic community now after publishing a paper consisting of
> little apart from the two collisions themselves).
(2) I am not concerned with cryptography here, merely sound engineering
tradeoffs and the avoidance of _pain_in_the_ass_ when we do see a
random collision, [NB the 2^69 is to 'cause a collision in SHA1' not the
odds against such a collision] ... (below)
>
> Please, let's talk about hash collisions responsibly. I posted earlier
> about the *actual computed probability* of finding two files with an
> SHA-1 collision before the sun goes supernova. It's 10^28 to 1 against.
> The recent cryptographic works has shown that there are certain
> situations where a decent amount of computer work (2^69 operations) can
> produce two sequences with the same hash, but these sequences are not
> freely chosen; they've got very specific structure. This attack does
> not apply to (effectively) random files sitting in a SCM.
> http://www.schneier.com/blog/archives/2005/02/sha1_broken.html
>
> That said, Linux's widespread use means that it may not be unimaginable
> for an attacker to devote this amount of resources to an attack, which
> would probably involve first committing some specially structured file
> to the SCM (but would Linus accept it?) and then silently corrupting
> said file via a SHA1 collision to toggle some bits (which would
> presumably Do Evil). Thus hashes other than SHA1 really ought to be
> considered...
>
> ..but the cryptographic community has not yet come to a conclusion on
> what the replacement ought to be. These attacks are so new that we
> don't really understand what it is about the structure of SHA1 which
> makes them possible, which makes it hard to determine which other hashes
> are similarly vulnerable. It will take time.
>
> I believe Linus has already stated on this list that his plan is to
> eventually provide a tool for bulk migration of an existing SHA1 git
> repository to a new hash type. Basically munging through the
> repository in bulk, replacing all the hashes. This seems a perfectly
> adequate strategy at the moment.
... [I say again, the problem here is NOT forgery of hashes, though SCO
like paranoia ...] ... but the hashes are a tiny part of the total
space, even for trivial patches, so that, providing _NOW_ for a longer
hash (and then why not use, say, SHA-256 for now as well) is prudent.
We do not want to revisit the plumbing, in the next 3-10 years, for 16
bytes per hash.
Finally I can do no more than quote Schneier:
"SHA-1 has been broken. Not a reduced-round version. Not a simplified
version. The real thing. ...
It's time for us all to migrate away from SHA-1.
Luckily, there are alternatives. The National Institute of Standards and
Technology already has standards for longer -- and harder to break --
hash functions: SHA-224, SHA-256, SHA-384, and SHA-512. They're already
government standards, and can already be used." and there are FOSS
implementations.
Or, put more simply by Jon Callas, PGP's CTO: "It's time to walk, but
not run, to the fire exits. You don't see smoke, but the fire alarms
have gone off." That's basically what he said last August [2004].
> --scott
>
> WASHTUB Panama Minister Moscow explosives KUGOWN hack Marxist LPMEDLEY
> genetic immediate radar SCRANTON COBRA JANE KGB Shoal Bay atomic Bejing
> ( http://cscott.net/ )
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
>
--
mit freundlichen Grüßen, Brian.
Dr. Brian O'Mahoney
Mobile +41 (0)79 334 8035 Email: omb@bluewin.ch
Bleicherstrasse 25, CH-8953 Dietikon, Switzerland
PGP Key fingerprint = 33 41 A2 DE 35 7C CE 5D F5 14 39 C9 6D 38 56 D5
^ permalink raw reply
* [PATCH] Get commits from remote repositories by HTTP
From: Daniel Barkalow @ 2005-04-16 22:03 UTC (permalink / raw)
To: git
This adds a program to download a commit, the trees, and the blobs in them
from a remote repository using HTTP. It skips anything you already have.
There are a number of improvements possible, to be done if this catches
on, including, significantly, checking if the response was correct (or
even not an error).
It makes fsck-cache and rev-tree give harmless warnings, because it
includes some code that should probably be shared with them in revision.h
Signed-Off-By: Daniel Barkalow <barkalow@iabervon.org>
Index: Makefile
===================================================================
--- ed4f6e454b40650b904ab72048b2f93a068dccc3/Makefile (mode:100644 sha1:b39b4ea37586693dd707d1d0750a9b580350ec50)
+++ a65375b46154c90e7499b7e76998d430cd9cd29d/Makefile (mode:100644 sha1:d41860aed161a14ca61e7b6c7f591f65928bd61f)
@@ -14,7 +14,7 @@
PROG= update-cache show-diff init-db write-tree read-tree commit-tree \
cat-file fsck-cache checkout-cache diff-tree rev-tree show-files \
- check-files ls-tree merge-tree
+ check-files ls-tree merge-tree http-get
all: $(PROG)
@@ -23,6 +23,9 @@
LIBS= -lssl -lz
+http-get:%:%.o read-cache.o
+ $(CC) $(CFLAGS) -o $@ $^ $(LIBS)
+
init-db: init-db.o
update-cache: update-cache.o read-cache.o
Index: http-get.c
===================================================================
--- /dev/null (tree:ed4f6e454b40650b904ab72048b2f93a068dccc3)
+++ a65375b46154c90e7499b7e76998d430cd9cd29d/http-get.c (mode:100644 sha1:6a36cfa079519a7a3ad5b1618be8711c5127b531)
@@ -0,0 +1,175 @@
+#include <sys/socket.h>
+#include <netdb.h>
+#include <netinet/in.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <string.h>
+#include <stdlib.h>
+#include "cache.h"
+#include "revision.h"
+#include <errno.h>
+
+static struct sockaddr_in sockad;
+static char *url;
+static char *base;
+
+static int target_url(char *target)
+{
+ char *name;
+ struct hostent *entry;
+ if (memcmp(target, "http://", 7))
+ return -1;
+ url = target;
+ base = strchr(target + 7, '/');
+ name = malloc(base - (target + 7) + 1);
+ memcpy(name, target + 7, base - (target + 7));
+ name[base - (target + 7)] = '\0';
+ printf("Connect to %s\n", name);
+ entry = gethostbyname(name);
+ memcpy(&sockad.sin_addr.s_addr,
+ &((struct in_addr *)entry->h_addr)->s_addr, 4);
+ sockad.sin_port = htons(80);
+ sockad.sin_family = AF_INET;
+}
+
+static int get_connection()
+{
+ int fd = socket(AF_INET, SOCK_STREAM, 0);
+ if (connect(fd, (struct sockaddr*) &sockad,
+ sizeof(struct sockaddr_in))) {
+ perror(url);
+ }
+ return fd;
+}
+
+static void release_connection(int fd) {
+ close(fd);
+}
+
+static int fetch(unsigned char *sha1)
+{
+ int header_end_posn = 0;
+ int local;
+ char *hex = sha1_to_hex(sha1);
+ char *filename = sha1_file_name(sha1);
+ char buffer[4096];
+ int fd;
+ struct stat st;
+
+ if (!stat(filename, &st)) {
+ return 0;
+ }
+
+ fd = get_connection();
+ if (fd < 0) {
+ return 1;
+ }
+
+ write(fd, "GET ", 4);
+ write(fd, base, strlen(base));
+ write(fd, "objects/", 8);
+ write(fd, hex, 2);
+ write(fd, "/", 1);
+ write(fd, hex + 2, 38);
+ write(fd, " HTTP/1.0\r\n", 11);
+ write(fd, "\r\n", 2);
+
+ local = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);
+
+ do {
+ int sz = read(fd, buffer, 4096);
+ if (!sz) {
+ break;
+ }
+ if (sz < 0) {
+ perror("Reading from connection");
+ unlink(filename);
+ close(local);
+ return 1;
+ }
+ if (header_end_posn < 4) {
+ int i = 0;
+ char *flag = "\r\n\r\n";
+ while (i < sz && header_end_posn < 4) {
+ if (buffer[i] == flag[header_end_posn]) {
+ header_end_posn++;
+ } else {
+ header_end_posn = 0;
+ }
+ i++;
+ }
+ if (i < sz) {
+ write(local, buffer + i, sz - i);
+ }
+ continue;
+ }
+ write(local, buffer, sz);
+ } while (1);
+
+ close(local);
+
+ release_connection(fd);
+ return 0;
+}
+
+static int process_tree(unsigned char *sha1)
+{
+ void *buffer;
+ unsigned long size;
+ char type[20];
+
+ buffer = read_sha1_file(sha1, type, &size);
+ if (!buffer)
+ return -1;
+ if (strcmp(type, "tree"))
+ return -1;
+ while (size) {
+ int len = strlen(buffer) + 1;
+ unsigned char *sha1 = buffer + len;
+ unsigned int mode;
+ int retval;
+
+ if (size < len + 20 || sscanf(buffer, "%o", &mode) != 1)
+ return -1;
+
+ buffer = sha1 + 20;
+ size -= len + 20;
+
+ retval = fetch(sha1);
+ if (retval)
+ return -1;
+
+ if (S_ISDIR(mode)) {
+ retval = process_tree(sha1);
+ if (retval)
+ return -1;
+ }
+ }
+ return 0;
+}
+
+static int process_commit(unsigned char *sha1)
+{
+ struct revision *rev = lookup_rev(sha1);
+ if (parse_commit_object(rev))
+ return -1;
+
+ fetch(rev->tree);
+ process_tree(rev->tree);
+ return 0;
+}
+
+int main(int argc, char **argv)
+{
+ char *commit_id = argv[1];
+ char *url = argv[2];
+
+ unsigned char sha1[20];
+
+ get_sha1_hex(commit_id, sha1);
+
+ target_url(url);
+
+ fetch(sha1);
+ return process_commit(sha1);
+}
Index: revision.h
===================================================================
--- ed4f6e454b40650b904ab72048b2f93a068dccc3/revision.h (mode:100664 sha1:28d0de3261a61f68e4e0948a25a416a515cd2e83)
+++ a65375b46154c90e7499b7e76998d430cd9cd29d/revision.h (mode:100664 sha1:523bde6e14e18bb0ecbded8f83ad4df93fc467ab)
@@ -24,6 +24,7 @@
unsigned int flags;
unsigned char sha1[20];
unsigned long date;
+ unsigned char tree[20];
struct parent *parent;
};
@@ -111,4 +112,29 @@
}
}
+static int parse_commit_object(struct revision *rev)
+{
+ if (!(rev->flags & SEEN)) {
+ void *buffer, *bufptr;
+ unsigned long size;
+ char type[20];
+ unsigned char parent[20];
+
+ rev->flags |= SEEN;
+ buffer = bufptr = read_sha1_file(rev->sha1, type, &size);
+ if (!buffer || strcmp(type, "commit"))
+ return -1;
+ get_sha1_hex(bufptr + 5, rev->tree);
+ bufptr += 46; /* "tree " + "hex sha1" + "\n" */
+ while (!memcmp(bufptr, "parent ", 7) &&
+ !get_sha1_hex(bufptr+7, parent)) {
+ add_relationship(rev, parent);
+ bufptr += 48; /* "parent " + "hex sha1" + "\n" */
+ }
+ //rev->date = parse_commit_date(bufptr);
+ free(buffer);
+ }
+ return 0;
+}
+
#endif /* REVISION_H */
^ permalink raw reply
* Re: [PATCH] Get commits from remote repositories by HTTP
From: Martin Mares @ 2005-04-16 22:17 UTC (permalink / raw)
To: Daniel Barkalow; +Cc: git
In-Reply-To: <Pine.LNX.4.21.0504161750020.30848-100000@iabervon.org>
Hello!
> This adds a program to download a commit, the trees, and the blobs in them
> from a remote repository using HTTP. It skips anything you already have.
Is it really necessary to write your own HTTP downloader? If so, is it
necessary to forget basic stuff like the "Host:" header? ;-)
If you feel that it should be optimized for speed, then at least use
persistent connections.
> + if (memcmp(target, "http://", 7))
> + return -1;
Can crash if the string is too short.
> + entry = gethostbyname(name);
> + memcpy(&sockad.sin_addr.s_addr,
> + &((struct in_addr *)entry->h_addr)->s_addr, 4);
Can crash if the host doesn't exist or if you feed it with an URL containing
port number.
> +static int get_connection()
(void)
> + local = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);
What if it fails?
Have a nice fortnight
--
Martin `MJ' Mares <mj@ucw.cz> http://atrey.karlin.mff.cuni.cz/~mj/
Faculty of Math and Physics, Charles University, Prague, Czech Rep., Earth
A student who changes the course of history is probably taking an exam.
^ permalink raw reply
* Re: [PATCH] Get commits from remote repositories by HTTP
From: Tony Luck @ 2005-04-16 22:24 UTC (permalink / raw)
To: Daniel Barkalow; +Cc: git
In-Reply-To: <Pine.LNX.4.21.0504161750020.30848-100000@iabervon.org>
On 4/16/05, Daniel Barkalow <barkalow@iabervon.org> wrote:
> + buffer = read_sha1_file(sha1, type, &size);
You never free this buffer.
It would also be nice if you saved "tree" objects in some temporary file
and did not install them until after you had fetched all the blobs and
trees that this tree references. Then if your connection is interrupted
you can just restart it.
Otherwise this looks really nice. I was going to script something
similar using "wget" ... but that would have made zillions of seperate
connections. Not so kind to the server.
-Tony
^ permalink raw reply
* Re: [PATCH] Get commits from remote repositories by HTTP
From: Jan-Benedict Glaw @ 2005-04-16 22:32 UTC (permalink / raw)
To: Daniel Barkalow; +Cc: git
In-Reply-To: <Pine.LNX.4.21.0504161750020.30848-100000@iabervon.org>
On Sat, 2005-04-16 18:03:51 -0400, Daniel Barkalow <barkalow@iabervon.org>
wrote in message <Pine.LNX.4.21.0504161750020.30848-100000@iabervon.org>:
> --- /dev/null (tree:ed4f6e454b40650b904ab72048b2f93a068dccc3)
> +++ a65375b46154c90e7499b7e76998d430cd9cd29d/http-get.c (mode:100644 sha1:6a36cfa079519a7a3ad5b1618be8711c5127b531)
> + local = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);
0666 is a bit too lazy. I'd suggest 0664 or 0644.
MfG, JBG
--
Jan-Benedict Glaw jbglaw@lug-owl.de . +49-172-7608481 _ O _
"Eine Freie Meinung in einem Freien Kopf | Gegen Zensur | Gegen Krieg _ _ O
fuer einen Freien Staat voll Freier Bürger" | im Internet! | im Irak! O O O
ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));
^ permalink raw reply
* Re: [PATCH] Get commits from remote repositories by HTTP
From: Daniel Barkalow @ 2005-04-16 22:33 UTC (permalink / raw)
To: Tony Luck; +Cc: git
In-Reply-To: <12c511ca050416152452a4c620@mail.gmail.com>
On Sat, 16 Apr 2005, Tony Luck wrote:
> On 4/16/05, Daniel Barkalow <barkalow@iabervon.org> wrote:
> > + buffer = read_sha1_file(sha1, type, &size);
>
> You never free this buffer.
Ideally, this should all be rearranged to share the code with
read-tree, and it should be fixed in common.
> It would also be nice if you saved "tree" objects in some temporary file
> and did not install them until after you had fetched all the blobs and
> trees that this tree references. Then if your connection is interrupted
> you can just restart it.
It looks over everything relevant, even if it doesn't need to download
anything, so it should work to continue if it stops in between.
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: SHA1 hash safety
From: David Lang @ 2005-04-16 22:33 UTC (permalink / raw)
To: omb; +Cc: Ingo Molnar, git
In-Reply-To: <4261132A.3090907@khandalf.com>
On Sat, 16 Apr 2005, Brian O'Mahoney wrote:
> Three points:
> (1) I _have_ seen real-life collisions with MD5, in the context of
> Document management systems containing ~10^6 ms-WORD documents.
> (2) The HMAC (ethernet-harware-address) of any interface _should_
> help to make a unique Id.
you want a unique ID that can be computed directly from the file contents.
what file integrety programa (ala tripwire) do is to use multiple
identification routines (aide uses MD4+MD5+filesize IIRC)
>
> David Lang wrote:
>> On Sat, 16 Apr 2005, Ingo Molnar wrote:
>>
>>> * David Lang <david.lang@digitalinsight.com> wrote:
>>>
>>>> this issue was raised a few days ago in the context of someone
>>>> tampering with the files and it was decided that the extra checks were
>>>> good enough to prevent this (at least for now), but what about
>>>> accidental collisions?
>>>>
>>>> if I am understanding things right the objects get saved in the
>>>> filesystem in filenames that are the SHA1 hash. of two legitimate
>>>> files have the same hash I don't see any way for both of them to
>>>> exist.
>>>>
>>>> yes the risk of any two files having the same has is low, but in the
>>>> earlier thread someone chimed in and said that they had two files on
>>>> their system that had the same hash..
>>>
>>>
>>> you can add -DCOLLISION_CHECK to Makefile:CFLAGS to turn on collision
>>> checking (disabled currently). If there indeed exist two files that have
>>> different content but the same hash, could someone send those two files?
>>
>>
>> remember that the flap over SHA1 being 'broken' a couple weeks ago was
>> not from researchers finding multiple files with the same hash, but
>> finding that it was more likly then expected that files would have the
>> same hash.
>>
>> there was qa discussion on LKML within the last year about useing MD5
>> hashes for identifying unique filesystem blocks (with the idea of being
>> able to merge identical blocks) and in that discussion it was pointed
>> out that collisions are a known real-life issue.
>>
>> so if collision detection is turned on in git, does that make it error
>> out if it runs into a second file with the same hash, or does it do
>> something else?
>>
>> David Lang
>>
>
> --
> Brian
>
--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare
^ permalink raw reply
* Re: [PATCH] Get commits from remote repositories by HTTP
From: Daniel Barkalow @ 2005-04-16 22:37 UTC (permalink / raw)
To: Jan-Benedict Glaw; +Cc: git
In-Reply-To: <20050416223206.GU9461@lug-owl.de>
On Sun, 17 Apr 2005, Jan-Benedict Glaw wrote:
> On Sat, 2005-04-16 18:03:51 -0400, Daniel Barkalow <barkalow@iabervon.org>
> wrote in message <Pine.LNX.4.21.0504161750020.30848-100000@iabervon.org>:
> > --- /dev/null (tree:ed4f6e454b40650b904ab72048b2f93a068dccc3)
> > +++ a65375b46154c90e7499b7e76998d430cd9cd29d/http-get.c (mode:100644 sha1:6a36cfa079519a7a3ad5b1618be8711c5127b531)
>
> > + local = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);
>
> 0666 is a bit too lazy. I'd suggest 0664 or 0644.
Actually, 0444 would make most sense, since these shouldn't get modified
at all. But umask is applied to them anyway, so 0664 or 0644 (or 0660 or
0600) is up to the local system policy. This just matches
write_sha1_buffer().
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: using git directory cache code in darcs?
From: Linus Torvalds @ 2005-04-16 22:43 UTC (permalink / raw)
To: David Roundy; +Cc: git, darcs-devel
In-Reply-To: <20050416132231.GJ2551@abridgegame.org>
On Sat, 16 Apr 2005, David Roundy wrote:
>
> I've been thinking about the possibility of using the git "current
> directory cache" code in darcs.
Go wild. The license is GPLv2, with the limitation that I really do want
to see v3 before I re-license anything at all, so if you take it into
darcs, you'd need to add that as a per-file comment (I just doing it in
the LICENSE file - I hate cluttering up individual files with tons of
commentary).
> So my questions are:
>
> 1) Would this actually be a good idea? It seems good to me, but there may
> be other considerations that I haven't thought of.
I really don't know how well the git index file will work with darcs, and
the main issue is that the index file names the "stable copy" using the
sha1 hash. If darcs uses something else (and I imagine it does) you'd need
to do a fair amount of surgery, and I suspect merging changes won't be
very easy.
So it might well make sense to wait a bit, until the git thing has calmed
down some more. For example, I made some rather large changes
(conceptually, if not in layout of the physical file) to the index file
just yesterday, since git now uses it for merging too.
In git, the index file isn't just a speedup, it's the "work" file _and_
the merge entity. It's not just a floor wax, it's a dessert topping too!
> 2) Will a license be chosen soon for git? Or has one been chosen, and I
> missed it? I can't really include git code in darcs without a license. I'd
> prefer GPLv2 or later (since that's how darcs is licensed), but as long as
> it's at least compabible with GPLv2, I'll be all right.
Yup, GPL, with the same "v2 by default" that the kernel uses).
> 3) Is it likely that git will switch to not using global variables for
> active_cache, active_nr and active_alloc?
I wouldn't hate it, although for the intent of git, the global approach
actually makes sense (dammit, I want the basic plumbing to be so small
that trying to abstract it out more is a waste of time). There's simply
not a lot of code that should even work at that level.
But if you wait a while, and bide your time, and then spring a clean patch
on me, I don't see any reason to be difficult about it either.
> 4) Would there be interest in creating a libgit? I've been imagining taking
> git source files and including them directly in darcs' code, but in the
> long run it would be easier if there were a standard git API we could use.
I think libgit might make sense, but again, not quite yet. Maybe the new
merge model was my last smart thought even on the subject of SCM's (I kind
of hope so), but maybe it's not.
My gut _feel_ is that the basic git low-level architecture is done, and
you can certainly start looking around and see if it matches darcs at all.
Linus
^ permalink raw reply
* Re: [PATCH] Get commits from remote repositories by HTTP
From: Adam Kropelin @ 2005-04-16 22:42 UTC (permalink / raw)
To: Tony Luck, Daniel Barkalow; +Cc: git
In-Reply-To: <12c511ca050416152452a4c620@mail.gmail.com>
Tony Luck wrote:
> Otherwise this looks really nice. I was going to script something
> similar using "wget" ... but that would have made zillions of seperate
> connections. Not so kind to the server.
How about building a file list and doing a batch download via 'wget -i
/tmp/foo'? A quick test (on my ancient wget-1.7) indicates that it reuses
connectionss when successive URLs point to the same server.
Writing yet another http client does seem a bit pointless, what with wget
and curl available. The real win lies in creating the smarts to get the
minimum number of files.
--Adam
^ permalink raw reply
* Re: [PATCH] Get commits from remote repositories by HTTP
From: Daniel Barkalow @ 2005-04-16 22:43 UTC (permalink / raw)
To: Martin Mares; +Cc: git
In-Reply-To: <20050416221745.GA10280@ucw.cz>
On Sun, 17 Apr 2005, Martin Mares wrote:
> Hello!
>
> > This adds a program to download a commit, the trees, and the blobs in them
> > from a remote repository using HTTP. It skips anything you already have.
>
> Is it really necessary to write your own HTTP downloader? If so, is it
> necessary to forget basic stuff like the "Host:" header? ;-)
I wanted to get something hacked quickly; can you suggest a good one to
use?
> If you feel that it should be optimized for speed, then at least use
> persistent connections.
That's the next step.
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: [PATCH] Get commits from remote repositories by HTTP
From: Daniel Barkalow @ 2005-04-16 22:45 UTC (permalink / raw)
To: Adam Kropelin; +Cc: Tony Luck, git
In-Reply-To: <011201c542d5$940bb670$03c8a8c0@kroptech.com>
On Sat, 16 Apr 2005, Adam Kropelin wrote:
> Tony Luck wrote:
> > Otherwise this looks really nice. I was going to script something
> > similar using "wget" ... but that would have made zillions of seperate
> > connections. Not so kind to the server.
>
> How about building a file list and doing a batch download via 'wget -i
> /tmp/foo'? A quick test (on my ancient wget-1.7) indicates that it reuses
> connectionss when successive URLs point to the same server.
You need to look at some of the files before you know what other files to
get. You could do it in waves, but that would be excessively complicated
to code and not the most efficient anyway.
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: SHA1 hash safety
From: David Lang @ 2005-04-16 22:46 UTC (permalink / raw)
To: C. Scott Ananian; +Cc: omb, Ingo Molnar, git
In-Reply-To: <Pine.LNX.4.61.0504161040310.29343@cag.csail.mit.edu>
that's the difference between CS researchers and sysadmins.
sysadmins realize that there are an infinante number of files that map to
the same hash value and plan accordingly (becouse we KNOW we will run
across them eventually), and don't see it as a big deal when we finally
do.
CS researches quote statistics that show how hard it is to intentiallly
create two files with the same hash and insist it just doesn't happen
until presented by the proof, at which point it is a big deal.
a difference in viewpoints.
David Lang
On Sat, 16 Apr 2005, C. Scott Ananian wrote:
> Date: Sat, 16 Apr 2005 10:58:15 -0400 (EDT)
> From: C. Scott Ananian <cscott@cscott.net>
> To: omb@bluewin.ch
> Cc: David Lang <david.lang@digitalinsight.com>, Ingo Molnar <mingo@elte.hu>,
> git@vger.kernel.org
> Subject: Re: SHA1 hash safety
>
> On Sat, 16 Apr 2005, Brian O'Mahoney wrote:
>
>> (1) I _have_ seen real-life collisions with MD5, in the context of
>> Document management systems containing ~10^6 ms-WORD documents.
>
> Dude! You could have been *famous*! Why the aitch-ee-double-hockey-sticks
> didn't you publish this when you found it?
> Seriously, man.
>
> Even given the known weaknesses in MD5, it would take much more than a
> million documents to find MD5 collisions. I can only conclude that the hash
> was being used incorrectly; most likely truncated (my wild-ass guess would be
> to 32 bits; a collision is likely with > 50% probability in a million
> document store for a hash of less than 40 bits).
>
> I know the current state of the art here. It's going to take more than just
> hearsay to convince me that full 128-bit MD5 collisions are likely. I believe
> there are only two or so known to exist so far, and those were found by a
> research team in China (which, yes, is fairly famous among the cryptographic
> community now after publishing a paper consisting of little apart from the
> two collisions themselves).
>
> Please, let's talk about hash collisions responsibly. I posted earlier about
> the *actual computed probability* of finding two files with an SHA-1
> collision before the sun goes supernova. It's 10^28 to 1 against.
> The recent cryptographic works has shown that there are certain situations
> where a decent amount of computer work (2^69 operations) can produce two
> sequences with the same hash, but these sequences are not freely chosen;
> they've got very specific structure. This attack does not apply to
> (effectively) random files sitting in a SCM.
> http://www.schneier.com/blog/archives/2005/02/sha1_broken.html
>
> That said, Linux's widespread use means that it may not be unimaginable for
> an attacker to devote this amount of resources to an attack, which would
> probably involve first committing some specially structured file to the SCM
> (but would Linus accept it?) and then silently corrupting said file via a
> SHA1 collision to toggle some bits (which would presumably Do Evil). Thus
> hashes other than SHA1 really ought to be considered...
>
> ...but the cryptographic community has not yet come to a conclusion on what
> the replacement ought to be. These attacks are so new that we don't really
> understand what it is about the structure of SHA1 which makes them possible,
> which makes it hard to determine which other hashes are similarly vulnerable.
> It will take time.
>
> I believe Linus has already stated on this list that his plan is to
> eventually provide a tool for bulk migration of an existing SHA1 git
> repository to a new hash type. Basically munging through the repository in
> bulk, replacing all the hashes. This seems a perfectly adequate strategy at
> the moment.
> --scott
>
> WASHTUB Panama Minister Moscow explosives KUGOWN hack Marxist LPMEDLEY
> genetic immediate radar SCRANTON COBRA JANE KGB Shoal Bay atomic Bejing
> ( http://cscott.net/ )
>
--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare
^ permalink raw reply
* [PATCH] update-cache --refresh cache entry leak
From: Junio C Hamano @ 2005-04-16 22:51 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
When update-cache --refresh replaces an existing cache entry
with a new one, it forgets to free the original.
Signed-off-by: Junio C Hamano <junkio@cox.net>
---
update-cache.c: 61d2b93a751f35ba24f479cd4fc151188916f02a
--- update-cache.c
+++ update-cache.c 2005-04-16 15:49:03.000000000 -0700
@@ -203,6 +203,8 @@
printf("%s: needs update\n", ce->name);
continue;
}
+ if (new != ce)
+ free(ce);
active_cache[i] = new;
}
}
^ permalink raw reply
* Full history
From: Thomas Gleixner @ 2005-04-16 23:52 UTC (permalink / raw)
To: git; +Cc: Linus Torvalds
Hi,
I can publish the stuff on monday from a university nearby.
---
total blob objects = 228384
total tree objects = 172507
total commit objects = 55877
The "empty" changesets which are noting merges are omitted at the
moment. Is it of interest to include them ??
It might also be interesting to export/merge the various
subsystem/maintainer trees including 2.4 into this archive. This would
cover the complete history
Disk space according to # du -sh
blobs ~ 2GiB
tree and commit objects ~ 1.3GiB
I looked at the spread of the 450k+ objects over the 256 subdirectories
in my exported git repository:
total 456768
max per XX subdir = 1646
avg per XX subdir = 1784
min per XX subdir = 1936
tglx
^ permalink raw reply
* Recent git pull troubles
From: Petr Baudis @ 2005-04-16 22:55 UTC (permalink / raw)
To: git
Hello,
just thought that I should drop a line here about it. I recently
changed the .git/HEAD to be a symlink, and old git pull could not cope
with that. So you would get a small error and no updates.
So far I fixed it by just making it a hardlink in my tree for now;
don't do that at home though, since git update will happily trash your
branch information in that case.
BTW, tomorrow will by my bugfix merging day (I will have a look
especially on Ingo Molnar's fixes), I will fix the bugs in git pull and
git update I mentioned in my night mail, and finally expand the docs;
sorry for the lack of activity on my side today. Then I will release 0.5
(getting 0.4 and doing git pull is not very smooth now), and after that
I will rewrite git merge to use Linus' merging stuff - can't wait to do
that. :-)
I'd still like you to constructively complain about my current branch
model. ;-)
Thanks,
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor
^ permalink raw reply
* Re: Re: SHA1 hash safety
From: David Lang @ 2005-04-16 22:56 UTC (permalink / raw)
To: C. Scott Ananian; +Cc: Petr Baudis, omb, Ingo Molnar, git
In-Reply-To: <Pine.LNX.4.61.0504161114530.29343@cag.csail.mit.edu>
On Sat, 16 Apr 2005, C. Scott Ananian wrote:
> Date: Sat, 16 Apr 2005 11:36:28 -0400 (EDT)
> From: C. Scott Ananian <cscott@cscott.net>
> To: Petr Baudis <pasky@ucw.cz>
> Cc: omb@bluewin.ch, David Lang <david.lang@digitalinsight.com>,
> Ingo Molnar <mingo@elte.hu>, git@vger.kernel.org
> Subject: Re: Re: SHA1 hash safety
>
> On Sat, 16 Apr 2005, Petr Baudis wrote:
>
>>> I know the current state of the art here. It's going to take more than
>>> just hearsay to convince me that full 128-bit MD5 collisions are likely.
>>
>> http://cryptography.hyperlink.cz/MD5_collisions.html
>
> OK, OK, I spoke too sloppily. Let me rephrase:
> It's going to take more than just hearsay to convince me that full
> 128-bit MD5 collisions *IN ARBITRARILY CHOSEN DOCUMENTS* are likely.
>
> I could add, "WITHOUT SPECIAL EFFORT BY AN ATTACKER".
you are missing the point.
I'm not talking about takeing one document (sched.c) and finding a
replacement that can drop in without being noticed.
what I'm talking about is the chance that somewhere, sometime there will
be two different documents that end up with the same hash
what git is doing (in very crude sysadminish terms) is to take all the
files you care about, move them into a new directory where they are named
by their hash with a symlink that replaces the origional file (and then a
bunch of stuff to manage multiple versions of those symlinks)
if you are taking every file that you ever care about and loosing all
refrence to it except by it's hash then when you get a second file that
has the same hash you loose the contents of one of the two files (race
condition over which file gets written into the storage directory last)
anywhere else that hashing algorithms are used people realize that there
will be hash collisions and plan accordingly, however people tend to put
blinders on when you say SHA1 or MD5 and decide that somehow the same
thing cannot happen with these types of hashes.
they can, and eventually they will so you need to plan accordingly.
David Lang
--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare
^ permalink raw reply
* Re-done kernel archive - real one?
From: Linus Torvalds @ 2005-04-16 23:01 UTC (permalink / raw)
To: Git Mailing List; +Cc: Peter Anvin
Ok, nobody really objected to the notion of leaving the kernel history
behind for now, and in fact most people seemed to basically agree. So with
that decided, the old kernel testing tree was actually perfectly ok,
except it had been build up with the old-style commit date handling, which
made me not want to use it as a base for any real work.
So I re-created the dang thing (hey, it takes just a few minutes), and
pushed it out, and there's now an archive on kernel.org in my public
"personal" directory called "linux-2.6.git". I'll continue the tradition
of naming git-archive directories as "*.git", since that really ends up
being the ".git" directory for the checked-out thing.
I'm not going to announce it on linux-kernel yet, because I don't think
it's useful to anybody but a git person anyway. Besides, I don't actually
know how happy the kernel.org people are about this distribution method
and whether it ends up being a horrible disaster for the mirroring setup.
Peter made some noises about /pub/scm, which makes sense, and would be a
better place than my public tree. Apparently there are other places that
are willing and able to host things too, so we'll see.
NOTE! The roughly 10x expansion of archive size goind from BK to git ends
up in a similar 10x bandwidth expansion, in addition to just the overhead
of reading tons of directory entries and comparing them (which is what
both a wget and rsync thing ends up doing). I'm sure we can bring that
down with smarter synchronization tools, but I also suspect that's some
way away.
So is real common usage, though, so maybe it's not that bad at all. Who
knows. We haven't hit a single real snag so far (except it took several
days longer than I expected, but hey, I expect lots of things ;), and I'm
sure real usage will show lots of them.
Similarly, we don't really have real merging, which makes tracking harder,
but I suspect actually having a tree out there will make people more
motivated and have more of a test-case. I'm feeling good enough about the
plumbing that I think I solved the "hard" part of it, and now it's just
the boring 95% left - scripting around it.
I think that with the new merge model, the easiest thing to do is to just
download all new objects, and then download the HEAD file under a new
name.
Ie we have two phases to the merge: first get the objects, with something
like
repo=kernel.org:/pub/kernel/people/torvalds/linux-2.6.git
rsync --ignore-existing -acv $(repo)/ .git/
which will _not_ download the new HEAD file (since you already have one of
your own), and then when you actually decide to merge you do
rsync -acv $(repo)/HEAD .git/MERGE_WITH
and now you can look at your old HEAD, and the MERGE_WITH thing, look up
the parents, and then do
read-tree -m <parent-tree> <head-tree> <merge-with-tree>
write-tree
commit-tree <result-tree> -p <head-tree> -p <merge-with-tree>
(which should actually _work_, assuming that the merge had no file
conflicts).
This seems to be a sane way to do merges, and if the scripting starts from
there and then becomes smarter...
Linus
^ permalink raw reply
* Re: Re: Add "clone" support to lntree
From: Petr Baudis @ 2005-04-16 23:00 UTC (permalink / raw)
To: Daniel Barkalow; +Cc: git
In-Reply-To: <Pine.LNX.4.21.0504152251300.30848-100000@iabervon.org>
Dear diary, on Sat, Apr 16, 2005 at 05:06:54AM CEST, I got a letter
where Daniel Barkalow <barkalow@iabervon.org> told me that...
> On Sat, 16 Apr 2005, Petr Baudis wrote:
> > I'm sorry but you are late, I added it about a hour and half ago or so.
> > :-) Check git fork. (I *want* separate command than git lntree. In fact,
> > I think I should make git lntree gitXlntree.sh instead, since it is
> > really internal command for git-tools and the user should probably never
> > need it for anything. git lntree is too lowlevel.)
>
> Have you not pushed since? I don't see it.
See my last mail. :-)
> I think "fork" is as good as anything for describing the operation. I had
> thought about "clone" because it seemed to fill the role that "bk
> clone" had (although I never used BK, so I'm not sure). It doesn't seem
> useful to me to try cloning multiple remote repositories, since you'd get
> a copy of anything common from each; you just want to suck everything into
> the same .git/objects and split off working directories.
Actually, what about if git pull outside of repository did what git
clone does now? I'd kinda like clone instead of fork too.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
C++: an octopus made by nailing extra legs onto a dog. -- Steve Taylor
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox