* Re: Howto update a 'dirty' entry in the cache from the object database
From: Linus Torvalds @ 2005-05-04 17:23 UTC (permalink / raw)
To: Thomas Glanzmann; +Cc: GIT
In-Reply-To: <20050504142351.GL18380@cip.informatik.uni-erlangen.de>
On Wed, 4 May 2005, Thomas Glanzmann wrote:
>
> I edited a bunch of files and notice that I want to revert back one of
> the files to the original one (the one that comes with HEAD). But I
> already updated the cache. Is there a way to 'patch' the cache with the
> original file.
Absolutely.
You can do it several ways. The easiest one is
git-read-tree -m HEAD
git-checkout-cache -f filename
but this means that you revert your entire index file to the old HEAD
information, so you'll need to do git-update-cache on the files that you
changed.
So:
> Or do I need todo a read-tree into a temporary index
> file. Check out the one file from this alternate index file and use
> update-cache to update my original index file?
Yes. If you want to touch just that one file, and you actually _want_ to
save all the other index information updates except for this one file. If
so, you can do
# read a new index file with the HEAD information
GIT_INDEX_FILE=tmp-index git-read-tree HEAD
# check out just the one file you want to have
GIT_INDEX_FILE=tmp-index git-checkout-cache -f filename
# remove the now useless temporary index
rm tmp-index
# update your _real_ index file with the file information
git-update-cache filename
and you're done - you've now updated only that _one_ file in your working
area, and you've re-set the index file for that entry.
And yes, scripting it so that you don't make stupid mistakes is probably a
good idea. For exmple, if you misspell GIT_INDEX_FILE (like I did when I
write that at first), you'd have reverted your real index file after all.
In other words, the git core certainly is very flexible and can easily do
what you want, but it's also certainly very very easy to screw up with.
Linus
^ permalink raw reply
* Re: [PATCH] Careful object pulling
From: Daniel Barkalow @ 2005-05-04 16:16 UTC (permalink / raw)
To: Morten Welinder; +Cc: Linus Torvalds, Git Mailing List
In-Reply-To: <118833cc050504023569e00d38@mail.gmail.com>
On Wed, 4 May 2005, Morten Welinder wrote:
> Something's fishy there. You are comparing the result from link with EEXIST.
No, it just looks that way. If ret is negative, errno gets written to
it. If it's zero, we don't do anything with it. It can't be positive. So,
at the point where we test it, it must be the errno from link, which would
be EEXIST if the case we're worried about.
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: [PATCH] add the ability to create and retrieve delta objects
From: C. Scott Ananian @ 2005-05-04 16:12 UTC (permalink / raw)
To: Chris Mason; +Cc: Nicolas Pitre, Linus Torvalds, Alon Ziv, git
In-Reply-To: <200505041156.19499.mason@suse.com>
On Wed, 4 May 2005, Chris Mason wrote:
> 3) create a git-pack tool that can pack/unpack existing changesets,trees and
> files, optionally adding/removing deltas.
A 'git-pull' tool might be more use. I can imagine Linus maintaining his
local tree uncompressed, but the 'kernel.org' tree set up to
git-pull-delta from him every hour or whatever, so that the
network-accessible version is always network-efficient. 'git-pack'
would then simplify to a git-pull-delta from an existing local repository.
Ideally, you'd also be able to git-pull from a network packed repository
and (transparently) unpack and undelta-fy the pulled files as they're
added to your local repo. This would keep Linus from accidentally getting
packed files in his tree when he pulled from a maintainer's
packed/delta-ed network-accessible tree.
I'd also be interested in seeing the speed/space numbers for some other
delta chain lengths between 1 and 16. Maybe some intermediate point is
optimal. [Also, limiting delta chains to a certain number of other
*packed* objects -- instead of just 'objects' -- might be an improvement.
Right now you're packing entire commits together, right? Maybe defining a
delta chain as 'N other commits max' might improve i/o performance, as
you'd just have to keep N other unpacked files around, instead of an
arbitrary number.]
--scott
SSBN 743 BOND ESGAIN SUMAC ZPSECANT MHCHAOS Castro Flintlock payment
anthrax SCRANTON PLO MKNAOMI DNC AVBLIMP RUFUS Secretary AK-47 Noriega
( http://cscott.net/ )
^ permalink raw reply
* Re: [PATCH] add the ability to create and retrieve delta objects
From: Chris Mason @ 2005-05-04 15:56 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Linus Torvalds, Alon Ziv, git
In-Reply-To: <Pine.LNX.4.62.0505030344170.14033@localhost.localdomain>
[-- Attachment #1: Type: text/plain, Size: 2655 bytes --]
On Tuesday 03 May 2005 04:06, Nicolas Pitre wrote:
> On Mon, 2 May 2005, Linus Torvalds wrote:
> > If you do something like this, you want such a delta-blob to be named by
> > the sha1 of the result, so that things that refer to it can transparently
> > see either the original blob _or_ the "deltified" one, and will never
> > care.
>
> Yep, that's what I've done last weekend (and just made it actually
> work since people are getting interested).
>
My first run didn't go well, diff_delta generates an invalid delta when passed
a buffer of length 0. I really should not have been calling it this way, but
it should do a quick check and return an error instead of something
invalid ;)
I did two additional runs, first where I fixed the delta chain length at 1 as
in the zdelta patch. In this mode, if it tried to diff against a delta it
would find the delta's parent and diff against that instead. Even though
zdelta had the same speeds for applying patches as xdiff(1), zdelta used
significantly more cpu (53m vs 40m).
The next run was with the patch I've attached below, it allows chains up to 16
deltas in length.
git zdelta xdiff (1) xdiff(16)
apply 150m 117m 117m 104m
checkout 4m30s 3m41 4m43s 7m11s
checkout (hot) 56s 12s 14s 16s
space usage 2.5G 1G 1.2G 800m
The longer delta chains trigger more random io on checkout, negating the speed
improvements from the packed item patch. The hot cache times show that xdiff
isn't using a huge amount of cpu to patch things in, and so there's room for
smarter packing and regenerating deltas in order to keep checkout times low.
This patch still doesn't pack commits and trees in with the blob files, and
it doesn't delta trees, and so I expect better space/speed numbers in later
revs.
I won't be able to work on this until next week, but here's my plan:
1) update to current git. My patch is from before the safe file generation
changes.
2) change update-cache and write-tree so that packing/deltas are off by
default. Add --packed and --delta options to both.
3) create a git-pack tool that can pack/unpack existing changesets,trees and
files, optionally adding/removing deltas.
My current code should preserve the delta object header used by Nicolas, and
removes all knowledge of deltas from the packed item headers. This is not
quite as efficient, but the resulting code is much cleaner. I haven't tried,
but it should be able to read a file created by his mkdelta.c.
-chris
[-- Attachment #2: delta-tree-3.diff --]
[-- Type: text/x-diff, Size: 32177 bytes --]
diff -urN --exclude .git linus.diff/cache.h linus.mine/cache.h
--- linus.diff/cache.h 2005-05-04 09:54:49.154735216 -0400
+++ linus.mine/cache.h 2005-05-04 08:47:28.618990016 -0400
@@ -64,6 +64,16 @@
char name[0];
};
+struct packed_item {
+ /* length of compressed data */
+ unsigned long len;
+ struct packed_item *next;
+ /* sha1 of uncompressed data */
+ char sha1[20];
+ /* compressed data */
+ char *data;
+};
+
#define CE_NAMEMASK (0x0fff)
#define CE_STAGEMASK (0x3000)
#define CE_STAGESHIFT 12
@@ -119,8 +129,9 @@
/* Read and unpack a sha1 file into memory, write memory to a sha1 file */
extern void * map_sha1_file(const unsigned char *sha1, unsigned long *size);
-extern void * unpack_sha1_file(void *map, unsigned long mapsize, char *type, unsigned long *size);
+extern void * unpack_sha1_file(const unsigned char *sha1, void *map, unsigned long mapsize, char *type, unsigned long *size, int *chain);
extern void * read_sha1_file(const unsigned char *sha1, char *type, unsigned long *size);
+extern void * read_sha1_delta_ref(const unsigned char *sha1, char *type, unsigned long *size, int *chain);
extern int write_sha1_file(char *buf, unsigned long len, const char *type, unsigned char *return_sha1);
extern int check_sha1_signature(unsigned char *sha1, void *buf, unsigned long size, const char *type);
@@ -135,6 +146,10 @@
/* Convert to/from hex/sha1 representation */
extern int get_sha1_hex(const char *hex, unsigned char *sha1);
extern char *sha1_to_hex(const unsigned char *sha1); /* static buffer result! */
+extern int pack_sha1_buffer(void *buf, unsigned long buf_len, char *type,
+ unsigned char *returnsha1, unsigned char *refsha1,
+ struct packed_item **);
+int write_packed_buffer(struct packed_item *head);
/* General helper functions */
extern void usage(const char *err);
diff -urN --exclude .git linus.diff/delta.h linus.mine/delta.h
--- linus.diff/delta.h 1969-12-31 19:00:00.000000000 -0500
+++ linus.mine/delta.h 2005-05-03 08:22:32.000000000 -0400
@@ -0,0 +1,6 @@
+extern void *diff_delta(void *from_buf, unsigned long from_size,
+ void *to_buf, unsigned long to_size,
+ unsigned long *delta_size);
+extern void *patch_delta(void *src_buf, unsigned long src_size,
+ void *delta_buf, unsigned long delta_size,
+ unsigned long *dst_size);
diff -urN --exclude .git linus.diff/diff-delta.c linus.mine/diff-delta.c
--- linus.diff/diff-delta.c 1969-12-31 19:00:00.000000000 -0500
+++ linus.mine/diff-delta.c 2005-05-03 12:40:58.000000000 -0400
@@ -0,0 +1,315 @@
+/*
+ * diff-delta.c: generate a delta between two buffers
+ *
+ * Many parts of this file have been lifted from LibXDiff version 0.10.
+ * http://www.xmailserver.org/xdiff-lib.html
+ *
+ * LibXDiff was written by Davide Libenzi <davidel@xmailserver.org>
+ * Copyright (C) 2003 Davide Libenzi
+ *
+ * Many mods for GIT usage by Nicolas Pitre <nico@cam.org>, (C) 2005.
+ *
+ * This file is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.1 of the License, or (at your option) any later version.
+ */
+
+#include <stdlib.h>
+#include "delta.h"
+
+
+/* block size: min = 16, max = 64k, power of 2 */
+#define BLK_SIZE 16
+
+#define MIN(a, b) ((a) < (b) ? (a) : (b))
+
+#define GR_PRIME 0x9e370001
+#define HASH(v, b) (((unsigned int)(v) * GR_PRIME) >> (32 - (b)))
+
+/* largest prime smaller than 65536 */
+#define BASE 65521
+
+/* NMAX is the largest n such that 255n(n+1)/2 + (n+1)(BASE-1) <= 2^32-1 */
+#define NMAX 5552
+
+#define DO1(buf, i) { s1 += buf[i]; s2 += s1; }
+#define DO2(buf, i) DO1(buf, i); DO1(buf, i + 1);
+#define DO4(buf, i) DO2(buf, i); DO2(buf, i + 2);
+#define DO8(buf, i) DO4(buf, i); DO4(buf, i + 4);
+#define DO16(buf) DO8(buf, 0); DO8(buf, 8);
+
+static unsigned int adler32(unsigned int adler, const unsigned char *buf, int len)
+{
+ int k;
+ unsigned int s1 = adler & 0xffff;
+ unsigned int s2 = adler >> 16;
+
+ while (len > 0) {
+ k = MIN(len, NMAX);
+ len -= k;
+ while (k >= 16) {
+ DO16(buf);
+ buf += 16;
+ k -= 16;
+ }
+ if (k != 0)
+ do {
+ s1 += *buf++;
+ s2 += s1;
+ } while (--k);
+ s1 %= BASE;
+ s2 %= BASE;
+ }
+
+ return (s2 << 16) | s1;
+}
+
+static unsigned int hashbits(unsigned int size)
+{
+ unsigned int val = 1, bits = 0;
+ while (val < size && bits < 32) {
+ val <<= 1;
+ bits++;
+ }
+ return bits ? bits: 1;
+}
+
+typedef struct s_chanode {
+ struct s_chanode *next;
+ int icurr;
+} chanode_t;
+
+typedef struct s_chastore {
+ chanode_t *head, *tail;
+ int isize, nsize;
+ chanode_t *ancur;
+ chanode_t *sncur;
+ int scurr;
+} chastore_t;
+
+static void cha_init(chastore_t *cha, int isize, int icount)
+{
+ cha->head = cha->tail = NULL;
+ cha->isize = isize;
+ cha->nsize = icount * isize;
+ cha->ancur = cha->sncur = NULL;
+ cha->scurr = 0;
+}
+
+static void *cha_alloc(chastore_t *cha)
+{
+ chanode_t *ancur;
+ void *data;
+
+ ancur = cha->ancur;
+ if (!ancur || ancur->icurr == cha->nsize) {
+ ancur = malloc(sizeof(chanode_t) + cha->nsize);
+ if (!ancur)
+ return NULL;
+ ancur->icurr = 0;
+ ancur->next = NULL;
+ if (cha->tail)
+ cha->tail->next = ancur;
+ if (!cha->head)
+ cha->head = ancur;
+ cha->tail = ancur;
+ cha->ancur = ancur;
+ }
+
+ data = (void *)ancur + sizeof(chanode_t) + ancur->icurr;
+ ancur->icurr += cha->isize;
+ return data;
+}
+
+static void cha_free(chastore_t *cha)
+{
+ chanode_t *cur = cha->head;
+ while (cur) {
+ chanode_t *tmp = cur;
+ cur = cur->next;
+ free(tmp);
+ }
+}
+
+typedef struct s_bdrecord {
+ struct s_bdrecord *next;
+ unsigned int fp;
+ const unsigned char *ptr;
+} bdrecord_t;
+
+typedef struct s_bdfile {
+ const unsigned char *data, *top;
+ chastore_t cha;
+ unsigned int fphbits;
+ bdrecord_t **fphash;
+} bdfile_t;
+
+static int delta_prepare(const unsigned char *buf, int bufsize, bdfile_t *bdf)
+{
+ unsigned int fphbits;
+ int i, hsize;
+ const unsigned char *base, *data, *top;
+ bdrecord_t *brec;
+ bdrecord_t **fphash;
+
+ fphbits = hashbits(bufsize / BLK_SIZE + 1);
+ hsize = 1 << fphbits;
+ fphash = malloc(hsize * sizeof(bdrecord_t *));
+ if (!fphash)
+ return -1;
+ for (i = 0; i < hsize; i++)
+ fphash[i] = NULL;
+ cha_init(&bdf->cha, sizeof(bdrecord_t), hsize / 4 + 1);
+
+ bdf->data = data = base = buf;
+ bdf->top = top = buf + bufsize;
+ data += (bufsize / BLK_SIZE) * BLK_SIZE;
+ if (data == top)
+ data -= BLK_SIZE;
+
+ for ( ; data >= base; data -= BLK_SIZE) {
+ brec = cha_alloc(&bdf->cha);
+ if (!brec) {
+ cha_free(&bdf->cha);
+ free(fphash);
+ return -1;
+ }
+ brec->fp = adler32(0, data, MIN(BLK_SIZE, top - data));
+ brec->ptr = data;
+ i = HASH(brec->fp, fphbits);
+ brec->next = fphash[i];
+ fphash[i] = brec;
+ }
+
+ bdf->fphbits = fphbits;
+ bdf->fphash = fphash;
+
+ return 0;
+}
+
+static void delta_cleanup(bdfile_t *bdf)
+{
+ free(bdf->fphash);
+ cha_free(&bdf->cha);
+}
+
+#define COPYOP_SIZE(o, s) \
+ (!!(o & 0xff) + !!(o & 0xff00) + !!(o & 0xff0000) + !!(o & 0xff000000) + \
+ !!(s & 0xff) + !!(s & 0xff00) + 1)
+
+void *diff_delta(void *from_buf, unsigned long from_size,
+ void *to_buf, unsigned long to_size,
+ unsigned long *delta_size)
+{
+ int i, outpos, outsize, inscnt, csize, msize, moff;
+ unsigned int fp;
+ const unsigned char *data, *top, *ptr1, *ptr2;
+ unsigned char *out, *orig;
+ bdrecord_t *brec;
+ bdfile_t bdf;
+
+ if (delta_prepare(from_buf, from_size, &bdf))
+ return NULL;
+
+ outpos = 0;
+ outsize = 4096;
+ out = malloc(outsize);
+ if (!out) {
+ delta_cleanup(&bdf);
+ return NULL;
+ }
+
+ data = to_buf;
+ top = to_buf + to_size;
+
+ out[outpos++] = from_size; from_size >>= 8;
+ out[outpos++] = from_size; from_size >>= 8;
+ out[outpos++] = from_size; from_size >>= 8;
+ out[outpos++] = from_size;
+ out[outpos++] = to_size; to_size >>= 8;
+ out[outpos++] = to_size; to_size >>= 8;
+ out[outpos++] = to_size; to_size >>= 8;
+ out[outpos++] = to_size;
+
+ inscnt = 0;
+ moff = 0;
+ while (data < top) {
+ msize = 0;
+ fp = adler32(0, data, MIN(top - data, BLK_SIZE));
+ i = HASH(fp, bdf.fphbits);
+ for (brec = bdf.fphash[i]; brec; brec = brec->next) {
+ if (brec->fp == fp) {
+ csize = bdf.top - brec->ptr;
+ if (csize > top - data)
+ csize = top - data;
+ for (ptr1 = brec->ptr, ptr2 = data;
+ csize && *ptr1 == *ptr2;
+ csize--, ptr1++, ptr2++);
+
+ csize = ptr1 - brec->ptr;
+ if (csize > msize) {
+ moff = brec->ptr - bdf.data;
+ msize = csize;
+ if (msize >= 0x10000) {
+ msize = 0x10000;
+ break;
+ }
+ }
+ }
+ }
+
+ if (!msize || msize < COPYOP_SIZE(moff, msize)) {
+ if (!inscnt)
+ outpos++;
+ out[outpos++] = *data++;
+ inscnt++;
+ if (inscnt == 0x7f) {
+ out[outpos - inscnt - 1] = inscnt;
+ inscnt = 0;
+ }
+ } else {
+ if (inscnt) {
+ out[outpos - inscnt - 1] = inscnt;
+ inscnt = 0;
+ }
+
+ data += msize;
+ orig = out + outpos++;
+ i = 0x80;
+
+ if (moff & 0xff) { out[outpos++] = moff; i |= 0x01; }
+ moff >>= 8;
+ if (moff & 0xff) { out[outpos++] = moff; i |= 0x02; }
+ moff >>= 8;
+ if (moff & 0xff) { out[outpos++] = moff; i |= 0x04; }
+ moff >>= 8;
+ if (moff & 0xff) { out[outpos++] = moff; i |= 0x08; }
+
+ if (msize & 0xff) { out[outpos++] = msize; i |= 0x10; }
+ msize >>= 8;
+ if (msize & 0xff) { out[outpos++] = msize; i |= 0x20; }
+
+ *orig = i;
+ }
+
+ /* next time around the largest possible output is 1 + 4 + 3 */
+ if (outpos > outsize - 8) {
+ void *tmp = out;
+ outsize = outsize * 3 / 2;
+ out = realloc(out, outsize);
+ if (!out) {
+ free(tmp);
+ delta_cleanup(&bdf);
+ return NULL;
+ }
+ }
+ }
+
+ if (inscnt)
+ out[outpos - inscnt - 1] = inscnt;
+
+ delta_cleanup(&bdf);
+ *delta_size = outpos;
+ return out;
+}
diff -urN --exclude .git linus.diff/fsck-cache.c linus.mine/fsck-cache.c
--- linus.diff/fsck-cache.c 2005-05-04 09:54:49.158734608 -0400
+++ linus.mine/fsck-cache.c 2005-05-04 08:44:49.778137496 -0400
@@ -142,7 +142,7 @@
if (map) {
char type[100];
unsigned long size;
- void *buffer = unpack_sha1_file(map, mapsize, type, &size);
+ void *buffer = unpack_sha1_file(sha1, map, mapsize, type, &size, NULL);
if (!buffer)
return -1;
if (check_sha1_signature(sha1, buffer, size, type) < 0)
diff -urN --exclude .git linus.diff/git-mktag.c linus.mine/git-mktag.c
--- linus.diff/git-mktag.c 2005-05-04 09:54:49.158734608 -0400
+++ linus.mine/git-mktag.c 2005-05-04 08:45:04.763859320 -0400
@@ -31,7 +31,7 @@
if (map) {
char type[100];
unsigned long size;
- void *buffer = unpack_sha1_file(map, mapsize, type, &size);
+ void *buffer = unpack_sha1_file(sha1, map, mapsize, type, &size, NULL);
if (buffer) {
if (!strcmp(type, expected_type))
diff -urN --exclude .git linus.diff/Makefile linus.mine/Makefile
--- linus.diff/Makefile 2005-05-04 09:54:49.153735368 -0400
+++ linus.mine/Makefile 2005-05-03 16:28:15.000000000 -0400
@@ -25,7 +25,8 @@
install: $(PROG) $(SCRIPTS)
install $(PROG) $(SCRIPTS) $(HOME)/bin/
-LIB_OBJS=read-cache.o sha1_file.o usage.o object.o commit.o tree.o blob.o
+LIB_OBJS=read-cache.o sha1_file.o usage.o object.o commit.o tree.o blob.o diff-delta.o \
+ patch-delta.o
LIB_FILE=libgit.a
LIB_H=cache.h object.h
diff -urN --exclude .git linus.diff/patch-delta.c linus.mine/patch-delta.c
--- linus.diff/patch-delta.c 1969-12-31 19:00:00.000000000 -0500
+++ linus.mine/patch-delta.c 2005-05-04 09:57:53.520707328 -0400
@@ -0,0 +1,80 @@
+/*
+ * patch-delta.c:
+ * recreate a buffer from a source and the delta produced by diff-delta.c
+ *
+ * (C) 2005 Nicolas Pitre <nico@cam.org>
+ *
+ * This code is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include "delta.h"
+
+void *patch_delta(void *src_buf, unsigned long src_size,
+ void *delta_buf, unsigned long delta_size,
+ unsigned long *dst_size)
+{
+ const unsigned char *data, *top;
+ unsigned char *dst, *out;
+ int size;
+
+ /* the smallest delta size possible is 10 bytes */
+ if (delta_size < 10) {
+ return NULL;
+ }
+ data = delta_buf;
+ top = delta_buf + delta_size;
+
+ /* make sure the orig file size matches what we expect */
+ size = data[0] | (data[1] << 8) | (data[2] << 16) | (data[3] << 24);
+ data += 4;
+ if (size != src_size) {
+ return NULL;
+ }
+ /* now the result size */
+ size = data[0] | (data[1] << 8) | (data[2] << 16) | (data[3] << 24);
+ data += 4;
+ dst = malloc(size);
+ if (!dst) {
+ return NULL;
+ }
+ out = dst;
+ while (data < top) {
+ unsigned char cmd = *data++;
+ if (cmd & 0x80) {
+ unsigned int cp_off = 0, cp_size = 0;
+ if (cmd & 0x01) cp_off = *data++;
+ if (cmd & 0x02) cp_off |= (*data++ << 8);
+ if (cmd & 0x04) cp_off |= (*data++ << 16);
+ if (cmd & 0x08) cp_off |= (*data++ << 24);
+ if (cmd & 0x10) cp_size = *data++;
+ if (cmd & 0x20) cp_size |= (*data++ << 8);
+ if (cp_size == 0) cp_size = 0x10000;
+ memcpy(out, src_buf + cp_off, cp_size);
+ out += cp_size;
+ if (out > dst + size) {
+ *(char *)0 = 0;
+ }
+ } else {
+ memcpy(out, data, cmd);
+ out += cmd;
+ data += cmd;
+ if (out > dst + size) {
+ *(char *)0 = 0;
+ }
+ }
+ }
+
+ /* sanity check */
+ if (data != top || out - dst != size) {
+ free(dst);
+ return NULL;
+ }
+
+ *dst_size = size;
+ return dst;
+}
diff -urN --exclude .git linus.diff/sha1_file.c linus.mine/sha1_file.c
--- linus.diff/sha1_file.c 2005-05-04 09:54:49.165733544 -0400
+++ linus.mine/sha1_file.c 2005-05-04 10:42:07.002316808 -0400
@@ -8,6 +8,7 @@
*/
#include <stdarg.h>
#include "cache.h"
+#include "delta.h"
const char *sha1_file_directory = NULL;
@@ -139,39 +140,117 @@
return map;
}
-void * unpack_sha1_file(void *map, unsigned long mapsize, char *type, unsigned long *size)
+/*
+ * looks through buf for the header entry corresponding to sha1. returns
+ * 0 an entry is found and sets offset to the offset of the packed item
+ * in the file. The offset is relative to the start of the packed items
+ * so you have to add in the length of the header before using it
+ * -1 is returned if the sha1 could not be found
+ */
+static int find_packed_header(const unsigned char *sha1, char *buf, unsigned long buf_len, unsigned long *offset)
+{
+ char *p;
+ p = buf;
+
+ *offset = 0;
+ while(p < buf + buf_len) {
+ unsigned long item_len;
+ unsigned char item_sha[20];
+ memcpy(item_sha, p, 20);
+ sscanf(p + 20, "%lu", &item_len);
+ p += 20 + strlen(p + 20) + 1;
+ if (memcmp(item_sha, sha1, 20) == 0)
+ return 0;
+ *offset += item_len;
+ }
+ return -1;
+}
+
+
+/*
+ * uncompresses a data segment without any extra delta/packed processing
+ */
+static void * _unpack_sha1_file(z_stream *stream, void *map,
+ unsigned long mapsize, char *type,
+ unsigned long *size)
{
int ret, bytes;
- z_stream stream;
char buffer[8192];
char *buf;
/* Get the data stream */
- memset(&stream, 0, sizeof(stream));
- stream.next_in = map;
- stream.avail_in = mapsize;
- stream.next_out = buffer;
- stream.avail_out = sizeof(buffer);
-
- inflateInit(&stream);
- ret = inflate(&stream, 0);
- if (ret < Z_OK)
+ memset(stream, 0, sizeof(*stream));
+ stream->next_in = map;
+ stream->avail_in = mapsize;
+ stream->next_out = buffer;
+ stream->avail_out = sizeof(buffer);
+
+ inflateInit(stream);
+ ret = inflate(stream, 0);
+ if (ret < Z_OK) {
return NULL;
- if (sscanf(buffer, "%10s %lu", type, size) != 2)
+ }
+ if (sscanf(buffer, "%10s %lu", type, size) != 2) {
return NULL;
-
+ }
bytes = strlen(buffer) + 1;
buf = xmalloc(*size);
- memcpy(buf, buffer + bytes, stream.total_out - bytes);
- bytes = stream.total_out - bytes;
+ memcpy(buf, buffer + bytes, stream->total_out - bytes);
+ bytes = stream->total_out - bytes;
if (bytes < *size && ret == Z_OK) {
- stream.next_out = buf + bytes;
- stream.avail_out = *size - bytes;
- while (inflate(&stream, Z_FINISH) == Z_OK)
+ stream->next_out = buf + bytes;
+ stream->avail_out = *size - bytes;
+ while (inflate(stream, Z_FINISH) == Z_OK)
/* nothing */;
}
- inflateEnd(&stream);
+ inflateEnd(stream);
+ return buf;
+}
+
+void * unpack_sha1_file(const unsigned char *sha1, void *map,
+ unsigned long mapsize, char *type, unsigned long *size,
+ int *chain)
+{
+ z_stream stream;
+ char *buf;
+ unsigned long offset;
+ unsigned long header_len;
+ buf = _unpack_sha1_file(&stream, map, mapsize, type, size);
+ if (!buf)
+ return buf;
+ if (!strcmp(type, "delta")) {
+ char *delta_ref;
+ unsigned long delta_size;
+ char *newbuf;
+ unsigned long newsize;
+ if (chain)
+ *chain += 1;
+ delta_ref = read_sha1_delta_ref(buf, type, &delta_size, chain);
+ if (!delta_ref) {
+ free(buf);
+ return NULL;
+ }
+ newbuf = patch_delta(delta_ref, delta_size, buf+20, *size-20, &newsize);
+ free(buf);
+ free(delta_ref);
+ *size = newsize;
+ return newbuf;
+
+ } else if (!strcmp(type, "packed")) {
+ if (!sha1) {
+ free(buf);
+ return NULL;
+ }
+ header_len = *size;
+ if (find_packed_header(sha1, buf, header_len, &offset)) {
+ free(buf);
+ return NULL;
+ }
+ offset += stream.total_in;
+ free(buf);
+ buf = unpack_sha1_file(sha1, map+offset, mapsize-offset, type, size, chain);
+ }
return buf;
}
@@ -182,7 +261,25 @@
map = map_sha1_file(sha1, &mapsize);
if (map) {
- buf = unpack_sha1_file(map, mapsize, type, size);
+ buf = unpack_sha1_file(sha1, map, mapsize, type, size, NULL);
+ munmap(map, mapsize);
+ return buf;
+ }
+ return NULL;
+}
+
+/*
+ * the same as read_sha1_file except chain is used to count the length
+ * of any delta chains hit while unpacking
+ */
+void * read_sha1_delta_ref(const unsigned char *sha1, char *type, unsigned long *size, int *chain)
+{
+ unsigned long mapsize;
+ void *map, *buf;
+
+ map = map_sha1_file(sha1, &mapsize);
+ if (map) {
+ buf = unpack_sha1_file(sha1, map, mapsize, type, size, chain);
munmap(map, mapsize);
return buf;
}
@@ -413,3 +510,306 @@
return 1;
return 0;
}
+
+static void *pack_buffer(void *buf, unsigned long buf_len, char *metadata, int metadata_size, unsigned long *compsize)
+{
+ char *compressed;
+ z_stream stream;
+ unsigned long size;
+
+ /* Set it up */
+ memset(&stream, 0, sizeof(stream));
+ size = deflateBound(&stream, buf_len + metadata_size);
+ compressed = xmalloc(size);
+
+ /*
+ * ASCII size + nul byte
+ */
+ stream.next_in = metadata;
+ stream.avail_in = metadata_size;
+ stream.next_out = compressed;
+ stream.avail_out = size;
+ deflateInit(&stream, Z_BEST_COMPRESSION);
+ while (deflate(&stream, 0) == Z_OK)
+ /* nothing */;
+
+ stream.next_in = buf;
+ stream.avail_in = buf_len;
+ /* Compress it */
+ while (deflate(&stream, Z_FINISH) == Z_OK)
+ /* nothing */;
+ deflateEnd(&stream);
+ size = stream.total_out;
+ *compsize = size;
+ return compressed;
+}
+
+/*
+ * generates a delta for buf against refsha1 and returns a compressed buffer
+ * with the results. NULL is returned on error, or when the delta could
+ * not be done. This might happen if the delta is larger then either the
+ * refsha1 or the buffer, or the delta chain is too long.
+ */
+static void *pack_delta_buffer(void *buf, unsigned long buf_len, char *metadata, int metadata_size, unsigned long *compsize, unsigned char *sha1, unsigned char *refsha1)
+{
+ char *compressed;
+ char *refbuffer = NULL;
+ char reftype[20];
+ unsigned long refsize = 0;
+ char *delta;
+ unsigned long delta_size;
+ char *lmetadata = xmalloc(220);
+ unsigned long lmetadata_size;
+ int chain_length = 0;
+
+ if (buf_len == 0)
+ return NULL;
+ refbuffer = read_sha1_delta_ref(refsha1, reftype, &refsize, &chain_length);
+
+ if (chain_length > 16) {
+ free(refbuffer);
+ return NULL;
+ }
+ /* note, we could just continue without the delta here */
+ if (!refbuffer) {
+ free(refbuffer);
+ return NULL;
+ }
+ delta = diff_delta(refbuffer, refsize, buf, buf_len, &delta_size);
+ free(refbuffer);
+ if (!delta)
+ return NULL;
+ if (delta_size > refsize || delta_size > buf_len) {
+ free(delta);
+ return NULL;
+ }
+ if (delta_size < 10) {
+ free(delta);
+ return NULL;
+ }
+ lmetadata_size = 1 + sprintf(lmetadata, "%s %lu","delta",delta_size+20);
+ memcpy(lmetadata + lmetadata_size, refsha1, 20);
+ lmetadata_size += 20;
+ compressed = pack_buffer(delta, delta_size, lmetadata, lmetadata_size, compsize);
+ free(lmetadata);
+ free(delta);
+ return compressed;
+}
+
+/*
+ * returns a newly malloc'd packed item with a compressed buffer for buf.
+ * If refsha1 is non-null, attempts a delta against it. The sha1 of buf
+ * is returned via returnsha1.
+ */
+int pack_sha1_buffer(void *buf, unsigned long buf_len, char *type,
+ unsigned char *returnsha1,
+ unsigned char *refsha1,
+ struct packed_item **packed_item)
+{
+ unsigned char sha1[20];
+ SHA_CTX c;
+ char *filename;
+ struct stat st;
+ char *compressed = NULL;
+ unsigned long size;
+ struct packed_item *item;
+ char *metadata = xmalloc(200);
+ int metadata_size;
+
+ *packed_item = NULL;
+
+ metadata_size = 1 + sprintf(metadata, "%s %lu", type, buf_len);
+
+ /* Sha1.. */
+ SHA1_Init(&c);
+ SHA1_Update(&c, metadata, metadata_size);
+ SHA1_Update(&c, buf, buf_len);
+ SHA1_Final(sha1, &c);
+
+ if (returnsha1)
+ memcpy(returnsha1, sha1, 20);
+
+ filename = sha1_file_name(sha1);
+ if (stat(filename, &st) == 0)
+ goto out;
+
+
+ if (refsha1) {
+ compressed = pack_delta_buffer(buf, buf_len, metadata,
+ metadata_size, &size, sha1,
+ refsha1);
+ }
+ if (!compressed) {
+ compressed = pack_buffer(buf, buf_len, metadata,
+ metadata_size, &size);
+ }
+ free(metadata);
+ if (!compressed) {
+ return -1;
+ }
+ item = xmalloc(sizeof(struct packed_item));
+ memcpy(item->sha1, sha1, 20);
+ item->len = size;
+ item->next = NULL;
+ item->data = compressed;
+ *packed_item = item;
+out:
+ return 0;
+}
+
+static char *create_packed_header(struct packed_item *head, unsigned long *size)
+{
+ char *metadata = NULL;
+ int metadata_size = 0;
+ *size = 0;
+ int entry_size = 0;
+
+ while(head) {
+ char *p;
+ metadata = realloc(metadata, metadata_size + 220);
+ if (!metadata)
+ return NULL;
+ p = metadata+metadata_size;
+ memcpy(p, head->sha1, 20);
+ p += 20;
+ entry_size = 1 + sprintf(p, "%lu", head->len);
+ metadata_size += entry_size + 20;
+ head = head->next;
+ }
+ *size = metadata_size;
+ return metadata;
+}
+
+#define WRITE_BUFFER_SIZE 8192
+static char write_buffer[WRITE_BUFFER_SIZE];
+static unsigned long write_buffer_len;
+
+static int c_write(int fd, void *data, unsigned int len)
+{
+ while (len) {
+ unsigned int buffered = write_buffer_len;
+ unsigned int partial = WRITE_BUFFER_SIZE - buffered;
+ if (partial > len)
+ partial = len;
+ memcpy(write_buffer + buffered, data, partial);
+ buffered += partial;
+ if (buffered == WRITE_BUFFER_SIZE) {
+ if (write(fd, write_buffer, WRITE_BUFFER_SIZE) != WRITE_BUFFER_SIZE)
+ return -1;
+ buffered = 0;
+ }
+ write_buffer_len = buffered;
+ len -= partial;
+ data += partial;
+ }
+ return 0;
+}
+
+static int c_flush(int fd)
+{
+ if (write_buffer_len) {
+ int left = write_buffer_len;
+ if (write(fd, write_buffer, left) != left)
+ return -1;
+ write_buffer_len = 0;
+ }
+ return 0;
+}
+
+/*
+ * creates a new packed file for all the items in head. hard links are
+ * made from the sha1 of all the items back to the packd file, and then
+ * the packed file is unlinked.
+ */
+int write_packed_buffer(struct packed_item *head)
+{
+ unsigned char sha1[20];
+ SHA_CTX c;
+ char *filename;
+ char *metadata = xmalloc(200);
+ char *header;
+ int metadata_size;
+ int fd;
+ int ret = 0;
+ unsigned long header_len;
+ struct packed_item *item;
+ char *compressed;
+ z_stream stream;
+ unsigned long size;
+
+ header = create_packed_header(head, &header_len);
+ metadata_size = 1+sprintf(metadata, "packed %lu", header_len);
+ /*
+ * the header contains the sha1 of each item, so we only sha1 the
+ * header
+ */
+ SHA1_Init(&c);
+ SHA1_Update(&c, metadata, metadata_size);
+ SHA1_Update(&c, header, header_len);
+ SHA1_Final(sha1, &c);
+
+ filename = strdup(sha1_file_name(sha1));
+ fd = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);
+ if (fd < 0) {
+ /* add collision check! */
+ if (errno != EEXIST) {
+ ret = -errno;
+ }
+ goto out_nofile;
+ }
+ /* compress just the header info */
+ memset(&stream, 0, sizeof(stream));
+ deflateInit(&stream, Z_BEST_COMPRESSION);
+ size = deflateBound(&stream, header_len + metadata_size);
+ compressed = xmalloc(size);
+
+ stream.next_in = metadata;
+ stream.avail_in = metadata_size;
+ stream.next_out = compressed;
+ stream.avail_out = size;
+ while (deflate(&stream, 0) == Z_OK)
+ /* nothing */;
+ stream.next_in = header;
+ stream.avail_in = header_len;
+ while (deflate(&stream, Z_FINISH) == Z_OK)
+ /* nothing */;
+ deflateEnd(&stream);
+ size = stream.total_out;
+
+ c_write(fd, compressed, size);
+ free(compressed);
+
+ item = head;
+ while(item) {
+ if (c_write(fd, item->data, item->len)) {
+ ret = -EIO;
+ goto out;
+ }
+ item = item->next;
+ }
+ if (c_flush(fd)) {
+ ret = -EIO;
+ goto out;
+ }
+ item = head;
+ while(item) {
+ char *item_file;
+ struct packed_item *next = item->next;
+ item_file = sha1_file_name(item->sha1);
+ if (link(filename, item_file) && errno != EEXIST) {
+ ret = -errno;
+ break;
+ }
+ free(item->data);
+ free(item);
+ item = next;
+ }
+ unlink(filename);
+out:
+ close(fd);
+out_nofile:
+ free(header);
+ free(metadata);
+ free(filename);
+ return ret;
+}
diff -urN --exclude .git linus.diff/update-cache.c linus.mine/update-cache.c
--- linus.diff/update-cache.c 2005-05-04 09:54:49.167733240 -0400
+++ linus.mine/update-cache.c 2005-05-02 20:51:32.000000000 -0400
@@ -31,55 +31,39 @@
return (unsigned long)ptr > (unsigned long)-1000L;
}
-static int index_fd(unsigned char *sha1, int fd, struct stat *st)
+static int index_fd(unsigned char *sha1, unsigned char *refsha1, int fd, struct stat *st, struct packed_item **head, struct packed_item **tail, unsigned long *packed_size, int *packed_nr)
{
- z_stream stream;
unsigned long size = st->st_size;
- int max_out_bytes = size + 200;
- void *out = xmalloc(max_out_bytes);
- void *metadata = xmalloc(200);
- int metadata_size;
void *in;
- SHA_CTX c;
+ int ret;
+ struct packed_item *new_item;
in = "";
if (size)
in = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);
close(fd);
- if (!out || (int)(long)in == -1)
+ if ((int)(long)in == -1) {
return -1;
-
- metadata_size = 1+sprintf(metadata, "blob %lu", size);
-
- SHA1_Init(&c);
- SHA1_Update(&c, metadata, metadata_size);
- SHA1_Update(&c, in, size);
- SHA1_Final(sha1, &c);
-
- memset(&stream, 0, sizeof(stream));
- deflateInit(&stream, Z_BEST_COMPRESSION);
-
- /*
- * ASCII size + nul byte
- */
- stream.next_in = metadata;
- stream.avail_in = metadata_size;
- stream.next_out = out;
- stream.avail_out = max_out_bytes;
- while (deflate(&stream, 0) == Z_OK)
- /* nothing */;
-
- /*
- * File content
- */
- stream.next_in = in;
- stream.avail_in = size;
- while (deflate(&stream, Z_FINISH) == Z_OK)
- /*nothing */;
-
- deflateEnd(&stream);
-
- return write_sha1_buffer(sha1, out, stream.total_out);
+ }
+ ret = pack_sha1_buffer(in, size, "blob", sha1, refsha1, &new_item);
+ if (new_item) {
+ if (*tail)
+ (*tail)->next = new_item;
+ *tail = new_item;
+ if (!*head)
+ *head = new_item;
+ *packed_size += new_item->len;
+ *packed_nr++;
+ if (*packed_size > (512 * 1024) || *packed_nr > 1024) {
+ write_packed_buffer(*head);
+ *head = NULL;
+ *tail = NULL;
+ *packed_size = 0;
+ *packed_nr = 0;
+ }
+ }
+ munmap(in, size);
+ return ret;
}
/*
@@ -102,12 +86,14 @@
ce->ce_size = htonl(st->st_size);
}
-static int add_file_to_cache(char *path)
+static int add_file_to_cache(char *path, struct packed_item **packed_head, struct packed_item **packed_tail, unsigned long *packed_size, int *packed_nr)
{
int size, namelen;
struct cache_entry *ce;
struct stat st;
int fd;
+ int pos;
+ unsigned char *refsha1 = NULL;
fd = open(path, O_RDONLY);
if (fd < 0) {
@@ -129,8 +115,12 @@
fill_stat_cache_info(ce, &st);
ce->ce_mode = create_ce_mode(st.st_mode);
ce->ce_flags = htons(namelen);
+ pos = cache_name_pos(ce->name, namelen);
+ if (pos >= 0)
+ refsha1 = active_cache[pos]->sha1;
- if (index_fd(ce->sha1, fd, &st) < 0)
+ if (index_fd(ce->sha1, refsha1, fd, &st, packed_head,
+ packed_tail, packed_size, packed_nr) < 0)
return -1;
return add_cache_entry(ce, allow_add);
@@ -311,6 +301,10 @@
int allow_options = 1;
static char lockfile[MAXPATHLEN+1];
const char *indexfile = get_index_file();
+ struct packed_item *packed_head = NULL;
+ struct packed_item *packed_tail = NULL;
+ unsigned long packed_size = 0;
+ int packed_nr = 0;
snprintf(lockfile, sizeof(lockfile), "%s.lock", indexfile);
@@ -362,8 +356,13 @@
fprintf(stderr, "Ignoring path %s\n", argv[i]);
continue;
}
- if (add_file_to_cache(path))
+ if (add_file_to_cache(path, &packed_head, &packed_tail, &packed_size, &packed_nr))
die("Unable to add %s to database", path);
+
+ }
+ if (packed_head) {
+ if (write_packed_buffer(packed_head))
+ die("write packed buffer failed");
}
if (write_cache(newfd, active_cache, active_nr) || rename(lockfile, indexfile))
die("Unable to write new cachefile");
diff -urN --exclude .git linus.diff/write-tree.c linus.mine/write-tree.c
--- linus.diff/write-tree.c 2005-05-04 09:54:49.167733240 -0400
+++ linus.mine/write-tree.c 2005-05-02 20:51:32.000000000 -0400
@@ -5,24 +5,13 @@
*/
#include "cache.h"
-static int check_valid_sha1(unsigned char *sha1)
-{
- char *filename = sha1_file_name(sha1);
- int ret;
-
- /* If we were anal, we'd check that the sha1 of the contents actually matches */
- ret = access(filename, R_OK);
- if (ret)
- perror(filename);
- return ret;
-}
-
-static int write_tree(struct cache_entry **cachep, int maxentries, const char *base, int baselen, unsigned char *returnsha1)
+static int write_tree(struct cache_entry **cachep, int maxentries, const char *base, int baselen, unsigned char *returnsha1, struct packed_item **head)
{
unsigned char subdir_sha1[20];
unsigned long size, offset;
char *buffer;
int nr;
+ struct packed_item *item;
/* Guess at some random initial size */
size = 8192;
@@ -50,7 +39,7 @@
if (dirname) {
int subdir_written;
- subdir_written = write_tree(cachep + nr, maxentries - nr, pathname, dirname-pathname+1, subdir_sha1);
+ subdir_written = write_tree(cachep + nr, maxentries - nr, pathname, dirname-pathname+1, subdir_sha1, head);
nr += subdir_written;
/* Now we need to write out the directory entry into this tree.. */
@@ -62,9 +51,6 @@
sha1 = subdir_sha1;
}
- if (check_valid_sha1(sha1) < 0)
- exit(1);
-
entrylen = pathlen - baselen;
if (offset + entrylen + 100 > size) {
size = alloc_nr(offset + entrylen + 100);
@@ -77,7 +63,11 @@
nr++;
} while (nr < maxentries);
- write_sha1_file(buffer, offset, "tree", returnsha1);
+ pack_sha1_buffer(buffer, offset, "tree", returnsha1, NULL, &item);
+ if (item) {
+ item->next = *head;
+ *head = item;
+ }
free(buffer);
return nr;
}
@@ -87,6 +77,7 @@
int i, unmerged;
int entries = read_cache();
unsigned char sha1[20];
+ struct packed_item *head = NULL;
if (entries <= 0)
die("write-tree: no cache contents to write");
@@ -107,8 +98,12 @@
die("write-tree: not able to write tree");
/* Ok, write it out */
- if (write_tree(active_cache, entries, "", 0, sha1) != entries)
+ if (write_tree(active_cache, entries, "", 0, sha1, &head) != entries)
die("write-tree: internal error");
+ if (head) {
+ if (write_packed_buffer(head))
+ die("write_packed_buffer error");
+ }
printf("%s\n", sha1_to_hex(sha1));
return 0;
}
^ permalink raw reply
* Re: git and symlinks as tracked content
From: David A. Wheeler @ 2005-05-04 15:48 UTC (permalink / raw)
To: H. Peter Anvin
Cc: Junio C Hamano, Linus Torvalds, Andreas Gal, Kay Sievers, git
In-Reply-To: <42780185.7010204@zytor.com>
Linus Torvalds wrote:
>Also, right now git will actually ignore most of the permission bits
too.
>We can change that, and make it a dynamic setting somewhere (some flag in
>a ".git/settings" file or something), but it does boil down to the fact
>that a software development tree tracker wants different things than
>something that tracks system settings.
...
>So if you want to track system files, right now "raw git" is _not_ the
way
>to do it. You'd want something else.
...
>But if you'd want to track other system directories with git, you'd
>probably need to either (a) do serious surgery on git itself, or (probably
>preferable) by (b) track the extra things you want "manually" using a file
>(that is tracked in git) that describes the ownership and permission data.
>
>Whether git is really suitable for tracking non-source projects is
>obviously debatable. It's not what it was designed for, and it _may_ be
>able to do so partly just by luck.
I suspect there's a 95% point which is easily achieved, &
beyond that it's not clear it's worth it.
I recall seeing several source code directories that actually use symlinks
in their source, and thus would want them preserved by the SCM.
(Not arguing that's the BEST plan, merely an observation).
As this discussion has noted, that wouldn't be hard to add symlink
support to git, and WOULD be helpful for its primary purpose as SCM support.
Once you're there, it wouldn't be hard to add logic to add options to
(1) record the REAL permission bits, (2) record "." files, and
(3) recover the permission bits. That would be enough to
store & recover in a distributed way a single person's home directory.
THAT might be darn useful, for those of us who float between
different systems & would like to use a single system for multiple purposes.
That's clearly beyond the scope of a typical SCM, but since
it's easy to get there, that'd make sense.
I'm ambivalent about supporting dev, uid/gid, and mtime, and how
it should be done; that may be beyond the "worth it" step.
--- David A. Wheeler
^ permalink raw reply
* Re: commit-id fails after cg-init
From: H. Peter Anvin @ 2005-05-04 15:45 UTC (permalink / raw)
To: David A. Wheeler; +Cc: Petr Baudis, Pavel Roskin, git, Joel.Becker
In-Reply-To: <4278E6D4.6060807@dwheeler.com>
David A. Wheeler wrote:
> Joel Becker said:
>
>> Well, cg-init in this case creates no objects. I'd say,
>> instead, it should create an empty tree object (representing a project
>> with no files) and commit that. That would be your initial commit, and
>> would put something valid in heads/master.
>
> That would actually make sense; commits would go all the way
> back to the "empty tree" as the ultimate initial tree.
>
> There's an interesting side-effect of this; I _think_ it's
> fine but it might be worth thinking through. If all
> new projects start with an empty tree, that creates a
> "common root" that all projects can appeal to.
> That means that in theory a merge between any two project root
> trees can eventually find a common ancestor: the empty tree.
> I _think_ that's okay... is it?
>
In fact, I think that's a Very Good Thing... it eliminates an
unnecessary corner case. Same reason linked lists want head nodes and
all that jazz.
-hpa
^ permalink raw reply
* Re: [PATCH 0/3] cogito spec file updates
From: Chris Wright @ 2005-05-04 15:27 UTC (permalink / raw)
To: H. Peter Anvin; +Cc: Petr Baudis, Chris Wright, Mark Allen, git
In-Reply-To: <4278DE7D.3000005@zytor.com>
* H. Peter Anvin (hpa@zytor.com) wrote:
> What I usually do is to have a *.spec.in file, and have my release
> script generate the *.spec file. I usually have a "version" file
> checked into the SCM from which all version information derives,
> including what to put in the *.spec file as well as what to name the
> subdirectory.
>
> The release script then ends up being some variant on:
>
> #!/bin/sh -xe
> PACKAGE=pkgname
> VERSION=`cat version`
> scm-of-choice tag --force $PACKAGE-$VERSION
> mkdir /var/tmp/$PACKAGE-$VERSION
> cd /var/tmp/$PACKAGE-$VERSION
> scm-of-choice export -r $PACKAGE-$VERSION
> make release
> cd ..
> tar cvvfz $PACKAGE-$VERSION.tar.gz $PACKAGE-$VERSION
> rm -rf $PACKAGE-$VERSION
>
> ... where "make release" creates the specfile and anything else that
> needs to be created (like autoconf files.)
Yup, that's precisely what I was thinking. I cobbled up an add-on to
Mark Allen's patch yesterday, but didn't send it since Pasky didn't seem
too interested in the idea. Anyway, here it is to demonstrate the idea.
The current setup is definitely workable, just means each release tarball
will build a bogus package with rpmbuild -ta. So I just fix up the spec
and then build...
thanks,
-chris
--
Autogen git.spec from git.spec.in. Needs patch from Mark Allen.
Signed-off-by: Chris Wright <chrisw@osdl.org>
--- cogito/Makefile~orig 2005-05-03 14:41:50.000000000 -0700
+++ cogito/Makefile 2005-05-03 15:53:36.000000000 -0700
@@ -126,6 +126,8 @@
@chmod +x $@
endif
+git.spec: git.spec.in $(VERSION)
+ sed -e 's/@@VERSION@@/$(shell cat $(VERSION) | cut -d"-" -f2)/g' < $< > $@
install: $(PROG) $(SCRIPTS) $(SCRIPT) $(GEN_SCRIPT)
install -m755 -d $(DESTDIR)$(bindir)
@@ -141,10 +143,10 @@
backup: clean
cd .. ; tar czvf dircache.tar.gz dir-cache
-release-tar-gzip: clean
+release-tar-gzip: clean git.spec
tar czf $(shell cat $(VERSION)).tar.gz *.c *.h git.spec Makefile $(DOCS) $(DIRS) $(SCRIPTS) $(SCRIPT)
-release-tar-bzip2: clean
+release-tar-bzip2: clean git.spec
tar cjf $(shell cat $(VERSION)).tar.bz2 *.c *.h git.spec Makefile $(DOCS) $(DIRS) $(SCRIPTS) $(SCRIPT)
release-tarballs: release-tar-gzip release-tar-bzip2
--- cogito/git.spec~orig 2005-05-03 14:41:59.000000000 -0700
+++ cogito/git.spec 2005-05-03 14:42:55.000000000 -0700
@@ -1,55 +0,0 @@
-Name: cogito
-Version: 0.8
-Release: 2
-Vendor: Petr Baudis <pasky@ucw.cz>
-Summary: Git core and tools
-License: GPL
-Group: Development/Tools
-URL: http://kernel.org/pub/software/scm/cogito/
-Source: http://kernel.org/pub/software/scm/cogito/%{name}-%{version}.tar.bz2
-Provides: cogito = %{version}
-Obsoletes: git
-BuildRequires: zlib-devel, openssl-devel, curl-devel
-BuildRoot: %{_tmppath}/%{name}-%{version}-root
-Prereq: sh-utils, diffutils, rsync, rcs, mktemp >= 1.5
-
-%description
-GIT comes in two layers. The bottom layer is merely an extremely fast
-and flexible filesystem-based database designed to store directory trees
-with regard to their history. The top layer is a SCM-like tool which
-enables human beings to work with the database in a manner to a degree
-similar to other SCM tools (like CVS, BitKeeper or Monotone).
-
-%prep
-%setup -q
-
-%build
-
-make
-
-%install
-rm -rf $RPM_BUILD_ROOT
-make DESTDIR=$RPM_BUILD_ROOT prefix=%{_prefix} install
-
-%clean
-rm -rf $RPM_BUILD_ROOT
-
-%files
-%defattr(-,root,root)
-/usr/bin/*
-%doc README README.reference COPYING Changelog
-
-%changelog
-* Wed Apr 27 2005 Terje Rosten <terje.rosten@ntnu.no> 0.8-2
-- Doc files
-- Use %%{_prefix} macro
-- Drop -n option to %%setup macro
-
-* Mon Apr 25 2005 Chris Wright <chrisw@osdl.org> 0.8-1
-- Update to cogito, rename package, move to /usr/bin, update prereqs
-
-* Mon Apr 25 2005 Chris Wright <chrisw@osdl.org> 0.7-1
-- Update to 0.7
-
-* Thu Apr 21 2005 Chris Wright <chrisw@osdl.org> 0.6.3-1
-- Initial rpm build
--- cogito/git.spec.in~orig 2005-05-03 14:42:20.000000000 -0700
+++ cogito/git.spec.in 2005-05-03 14:44:19.000000000 -0700
@@ -0,0 +1,58 @@
+Name: cogito
+Version: @@VERSION@@
+Release: 1
+Vendor: Petr Baudis <pasky@ucw.cz>
+Summary: Git core and tools
+License: GPL
+Group: Development/Tools
+URL: http://kernel.org/pub/software/scm/cogito/
+Source: http://kernel.org/pub/software/scm/cogito/%{name}-%{version}.tar.bz2
+Provides: cogito = %{version}
+Obsoletes: git
+BuildRequires: zlib-devel, openssl-devel, curl-devel
+BuildRoot: %{_tmppath}/%{name}-%{version}-root
+Prereq: sh-utils, diffutils, rsync, rcs, mktemp >= 1.5
+
+%description
+GIT comes in two layers. The bottom layer is merely an extremely fast
+and flexible filesystem-based database designed to store directory trees
+with regard to their history. The top layer is a SCM-like tool which
+enables human beings to work with the database in a manner to a degree
+similar to other SCM tools (like CVS, BitKeeper or Monotone).
+
+%prep
+%setup -q
+
+%build
+
+make
+
+%install
+rm -rf $RPM_BUILD_ROOT
+make DESTDIR=$RPM_BUILD_ROOT prefix=%{_prefix} install
+
+%clean
+rm -rf $RPM_BUILD_ROOT
+
+%files
+%defattr(-,root,root)
+/usr/bin/*
+%doc README README.reference COPYING Changelog
+
+%changelog
+* Tue May 3 2005 Chris Wright <chrisw@osdl.org>
+- Auto build from git.spec.in
+
+* Wed Apr 27 2005 Terje Rosten <terje.rosten@ntnu.no> 0.8-2
+- Doc files
+- Use %%{_prefix} macro
+- Drop -n option to %%setup macro
+
+* Mon Apr 25 2005 Chris Wright <chrisw@osdl.org> 0.8-1
+- Update to cogito, rename package, move to /usr/bin, update prereqs
+
+* Mon Apr 25 2005 Chris Wright <chrisw@osdl.org> 0.7-1
+- Update to 0.7
+
+* Thu Apr 21 2005 Chris Wright <chrisw@osdl.org> 0.6.3-1
+- Initial rpm build
^ permalink raw reply
* Re: commit-id fails after cg-init
From: David A. Wheeler @ 2005-05-04 15:14 UTC (permalink / raw)
To: Petr Baudis; +Cc: Pavel Roskin, git, Joel.Becker
In-Reply-To: <20050503211301.GA15995@pasky.ji.cz>
Joel Becker said:
> Well, cg-init in this case creates no objects. I'd say,
>instead, it should create an empty tree object (representing a project
>with no files) and commit that. That would be your initial commit, and
>would put something valid in heads/master.
That would actually make sense; commits would go all the way
back to the "empty tree" as the ultimate initial tree.
There's an interesting side-effect of this; I _think_ it's
fine but it might be worth thinking through. If all
new projects start with an empty tree, that creates a
"common root" that all projects can appeal to.
That means that in theory a merge between any two project root
trees can eventually find a common ancestor: the empty tree.
I _think_ that's okay... is it?
That also means that empty directories will end up with the
"empty tree" as well. Is there a risk of multiple empty directories
causing problems later? As far as I can tell, there aren't
any problems with that, and does seem logically sound.
--- David A. Wheeler
^ permalink raw reply
* Re: [PATCH 0/3] cogito spec file updates
From: H. Peter Anvin @ 2005-05-04 15:01 UTC (permalink / raw)
To: Horst von Brand; +Cc: Petr Baudis, Chris Wright, Mark Allen, git
In-Reply-To: <200505041454.j44Eslpg004032@laptop11.inf.utfsm.cl>
Horst von Brand wrote:
> "H. Peter Anvin" <hpa@zytor.com> said:
>
>>Petr Baudis wrote:
>>
>>>I wouldn't accept this neither. If git.spec is already version
>>>controlled, it should be up-to-date in the version control. Therefore,
>>>you need to update it at the time of release, not at the time of
>>>generating the tarball.
>
>
>>What I usually do is to have a *.spec.in file, and have my release
>>script generate the *.spec file. I usually have a "version" file
>>checked into the SCM from which all version information derives,
>>including what to put in the *.spec file as well as what to name the
>>subdirectory.
>
> Right. Note that this is /not/ autoconfiguring .spec from .spec.in, you are
> "just" automating the process I do by hand.
Right, the .spec.in here doesn't imply the use of autoconf.
(That being said, if you *are* using autoconf, the "make release" target
is also a good place to produce the autoconf generated files, being just
another class of autogenerated-but-should-be-in-the-tarball files.)
-hpa
^ permalink raw reply
* Re: [PATCH 0/3] cogito spec file updates
From: Horst von Brand @ 2005-05-04 14:54 UTC (permalink / raw)
To: H. Peter Anvin; +Cc: Petr Baudis, Chris Wright, Mark Allen, git
In-Reply-To: <4278DE7D.3000005@zytor.com>
"H. Peter Anvin" <hpa@zytor.com> said:
> Petr Baudis wrote:
> > I wouldn't accept this neither. If git.spec is already version
> > controlled, it should be up-to-date in the version control. Therefore,
> > you need to update it at the time of release, not at the time of
> > generating the tarball.
> What I usually do is to have a *.spec.in file, and have my release
> script generate the *.spec file. I usually have a "version" file
> checked into the SCM from which all version information derives,
> including what to put in the *.spec file as well as what to name the
> subdirectory.
Right. Note that this is /not/ autoconfiguring .spec from .spec.in, you are
"just" automating the process I do by hand.
--
Dr. Horst H. von Brand User #22616 counter.li.org
Departamento de Informatica Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria +56 32 654239
Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
^ permalink raw reply
* Re: [PATCH 0/3] cogito spec file updates
From: H. Peter Anvin @ 2005-05-04 14:38 UTC (permalink / raw)
To: Petr Baudis; +Cc: Chris Wright, Mark Allen, git
In-Reply-To: <20050503214401.GE15995@pasky.ji.cz>
Petr Baudis wrote:
>
> I wouldn't accept this neither. If git.spec is already version
> controlled, it should be up-to-date in the version control. Therefore,
> you need to update it at the time of release, not at the time of
> generating the tarball.
>
What I usually do is to have a *.spec.in file, and have my release
script generate the *.spec file. I usually have a "version" file
checked into the SCM from which all version information derives,
including what to put in the *.spec file as well as what to name the
subdirectory.
The release script then ends up being some variant on:
#!/bin/sh -xe
PACKAGE=pkgname
VERSION=`cat version`
scm-of-choice tag --force $PACKAGE-$VERSION
mkdir /var/tmp/$PACKAGE-$VERSION
cd /var/tmp/$PACKAGE-$VERSION
scm-of-choice export -r $PACKAGE-$VERSION
make release
cd ..
tar cvvfz $PACKAGE-$VERSION.tar.gz $PACKAGE-$VERSION
rm -rf $PACKAGE-$VERSION
... where "make release" creates the specfile and anything else that
needs to be created (like autoconf files.)
-hpa
^ permalink raw reply
* Howto update a 'dirty' entry in the cache from the object database
From: Thomas Glanzmann @ 2005-05-04 14:23 UTC (permalink / raw)
To: GIT
Hello,
I edited a bunch of files and notice that I want to revert back one of
the files to the original one (the one that comes with HEAD). But I
already updated the cache. Is there a way to 'patch' the cache with the
original file. Or do I need todo a read-tree into a temporary index
file. Check out the one file from this alternate index file and use
update-cache to update my original index file?
Thomas [ who want's a "git revert" ]
^ permalink raw reply
* Re: [PATCH 0/3] cogito spec file updates
From: Horst von Brand @ 2005-05-04 1:00 UTC (permalink / raw)
To: Chris Wright; +Cc: Petr Baudis, git
In-Reply-To: <20050503193536.GE5324@shell0.pdx.osdl.net>
Chris Wright <chrisw@osdl.org> said:
> * Chris Wright (chrisw@osdl.org) wrote:
> > Here's the outstanding updates for the spec file, up to 0.8-2 which is
> > the latest on kernel.org.
> >
> > http://www.kernel.org/pub/software/scm/cogito/RPMS/
>
> What's your method for creating a release tarball? If it were formalized
> (i.e. Makefile rule), then it'd be simple to use VERSION to drive the
> spec file, and it'd only need updating for real content changes (similar
> to what Kay did).
In each case you should add a Changelog entry to the spec file. Said entry
will probably mention the version anyway. Updating the version by hand
while at it is no big deal, now is it? Probably even less hassle than doing
it automatically.
--
Dr. Horst H. von Brand User #22616 counter.li.org
Departamento de Informatica Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria +56 32 654239
Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
^ permalink raw reply
* [PATCH] Add optional destination to cg-clone.
From: Anton Altaparmakov @ 2005-05-04 12:00 UTC (permalink / raw)
To: Petr Baudis; +Cc: git
Hi Petr,
Below is a patch to latest cogito.git which adds support for an optional
destination to the cg-clone command, e.g. I like to use:
cd /usr/src
cg-clone rsync://blah/linux-2.6.git
cg-clone /usr/src/linux-2.6.git ntfs-2.6.git
Of course you can instead do:
cg-clone rsync://blah/linux-2.6.git /usr/src/linux-2.6.git
cg-clone /usr/src/linux-2.6.git /usr/src/ntfs-2.6.git
Without my patch you need to do:
cd /usr/src
cg-clone rsync://blah/linux-2.6.git
mkdir tmp.git
cd tmp.git
cg-clone /usr/src/linux-2.6.git
mv linux-2.6.git ../ntfs-2.6.git
cd ..
rmdir tmp.git
For some strange reason I prefer my version. (-;
Please apply. Thanks.
Signed-off-by: Anton Altaparmakov <aia21@cantab.net>
Best regards,
Anton
--
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/
---
--- cogito.git/cg-help.old 2005-05-04 12:35:35.000000000 +0100
+++ cogito.git/cg-help 2005-05-04 12:35:37.000000000 +0100
@@ -26,7 +26,7 @@ Available commands:
cg-branch-add BNAME SOURCE_LOC
cg-branch-ls
cg-cancel
- cg-clone SOURCE_LOC
+ cg-clone SOURCE_LOC [DESTINATION]
cg-commit [FILE]... < log message on stdin
cg-diff [-p] [-r FROM_ID[:TO_ID]] [FILE]...
cg-export DESTDIR [TREE_ID]
--- cogito.git/cg-clone.old 2005-05-03 21:33:05.000000000 +0100
+++ cogito.git/cg-clone 2005-05-04 12:43:04.000000000 +0100
@@ -3,18 +3,26 @@
# Clone a remote GIT repository.
# Copyright (c) Petr Baudis, 2005
#
-# This is like cg-init, but works only for remote repositories and
-# will create a new directory where it will do the checkout.
+# This is like cg-init, but it will create a new directory where it will do
+# the checkout.
#
-# Takes an parameter specifying location of the source repository.
+# Takes a parameter specifying the location of the source repository and an
+# optional second parameter specifying the destination. If the second
+# parameter is omitted, the basename of the source repository is used as the
+# destination.
. cg-Xlib
location=$1
-[ "$location" ] || die "usage: cg-clone SOURCE_LOC"
+[ "$location" ] || die "usage: cg-clone SOURCE_LOC [DESTINATION]"
location=${location%/}
-dir=${location%/.git}; dir=${dir##*/}
+destination=$2
+if [ "$destination" ]; then
+ dir=$destination
+else
+ dir=${location%/.git}; dir=${dir##*/}
+fi
[ -e "$dir" ] && die "$dir/ already exists"
mkdir "$dir" || exit $?
cd "$dir" || exit $?
^ permalink raw reply
* [PATCH] control/limit output of git-rev-list
From: Kay Sievers @ 2005-05-04 11:58 UTC (permalink / raw)
To: git; +Cc: Petr Baudis
gitweb.cgi's default view is the log of the last day and git-rev-list
can stop crawling the whole repo if we have all our data to display in the
browser. Also the rss-feed query needs only the last 20 items. This
will speeds up these queries dramatically.
usage: rev-list [OPTION] commit-id
--max-count=nr
--max-age=epoch
--min-age=epoch
Signed-off-by: Kay Sievers <kay.sievers@vrfy.org>
---
Pasky, kernel.org uses the cogito binaries. It would be nice, if you can
include this into the next release, so we can safe these boxes a whole lot
of useless work.
Thanks,
Kay
--- a/rev-list.c
+++ b/rev-list.c
@@ -6,9 +6,31 @@ int main(int argc, char **argv)
unsigned char sha1[20];
struct commit_list *list = NULL;
struct commit *commit;
+ char *commit_arg = NULL;
+ int i;
+ unsigned long max_age = -1;
+ unsigned long min_age = -1;
+ int max_count = -1;
- if (argc != 2 || get_sha1(argv[1], sha1))
- usage("rev-list <commit-id>");
+ for (i = 1 ; i < argc; i++) {
+ char *arg = argv[i];
+
+ if (!strncmp(arg, "--max-count=", 12)) {
+ max_count = atoi(arg + 12);
+ } else if (!strncmp(arg, "--max-age=", 10)) {
+ max_age = atoi(arg + 10);
+ } else if (!strncmp(arg, "--min-age=", 10)) {
+ min_age = atoi(arg + 10);
+ } else {
+ commit_arg = arg;
+ }
+ }
+
+ if (!commit_arg || get_sha1(commit_arg, sha1))
+ usage("usage: rev-list [OPTION] commit-id\n"
+ " --max-count=nr\n"
+ " --max-age=epoch\n"
+ " --min-age=epoch\n");
commit = lookup_commit(sha1);
if (!commit || parse_commit(commit) < 0)
@@ -17,6 +39,13 @@ int main(int argc, char **argv)
commit_list_insert(commit, &list);
do {
struct commit *commit = pop_most_recent_commit(&list, 0x1);
+
+ if (min_age != -1 && (commit->date > min_age))
+ continue;
+ if (max_age != -1 && (commit->date < max_age))
+ break;
+ if (max_count != -1 && !max_count--)
+ break;
printf("%s\n", sha1_to_hex(commit->object.sha1));
} while (list);
return 0;
^ permalink raw reply
* Re: [PATCH] Careful object pulling
From: Morten Welinder @ 2005-05-04 9:35 UTC (permalink / raw)
To: Daniel Barkalow; +Cc: Linus Torvalds, Git Mailing List
In-Reply-To: <Pine.LNX.4.21.0505040004290.30848-100000@iabervon.org>
Something's fishy there. You are comparing the result from link with EEXIST.
Morten
^ permalink raw reply
* plumbing pull request
From: Junio C Hamano @ 2005-05-04 9:22 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
Linus, please pull from http://members.cox.net/junkio/git-jc.git/
which contains the following changes.
----------------------------------------------------------------
Optimize diff-cache -p --cached
This patch optimizes "diff-cache -p --cached" by avoiding to
inflate blobs into temporary files when the blob recorded in the
cache matches the corresponding file in the work tree. The file
in the work tree is passed as the comparison source in such a
case instead.
This optimization kicks in only when we have already read the
cache this optimization and this is deliberate. Especially,
diff-tree does not use this code, because changes are contained
in small number of files relative to the project size most of
the time, and reading cache is so expensive for a large project
that the cost of reading it outweighs the savings by not
inflating blobs.
Also this patch cleans up the structure passed from diff clients
by removing one unused structure member.
----------------------------------------------------------------
Terminate diff-* on non-zero exit from GIT_EXTERNAL_DIFF
(slightly updated from the version posted to the GIT mailing list
with small bugfixes).
This patch changes the git-apply-patch-script to exit non-zero when
the patch cannot be applied. Previously, the external diff driver
deliberately ignored the exit status of GIT_EXTERNAL_DIFF command,
which was a design mistake. It now stops the processing when
GIT_EXTERNAL_DIFF exits non-zero, so the damages from running
git-diff-* with git-apply-patch-script between two wrong trees can be
contained.
The "diff" command line generated by the built-in driver is changed to
always exit 0 in order to match this new behaviour. I know Pasky does
not use GIT_EXTERNAL_DIFF yet, so this change should not break Cogito,
either.
----------------------------------------------------------------
Git-prune-script loses blobs referenced from an uncommitted cache.
(updated from the version posted to GIT mailing list).
When a new blob is registered with update-cache, and before the cache
is written as a tree and committed, git-fsck-cache will find the blob
unreachable. This patch adds a new flag, "--cache" to git-fsck-cache,
with which it keeps such blobs from considered "unreachable".
The git-prune-script is updated to use this new flag. At the same time
it adds .git/refs/*/* to the set of default locations to look for heads,
which should be consistent with expectations from Cogito users.
Without this fix, "diff-cache -p --cached" after git-prune-script has
pruned the blob object will fail mysteriously and git-write-tree would
also fail.
----------------------------------------------------------------
Short-cut error return path in git-local-pull.
When git-local-pull with -l option gets ENOENT attempting to create
a hard link, there is no point falling back to other copy methods.
With this patch, git-local-pull detects such a case and gives up
copying the file early.
----------------------------------------------------------------
Make git-*-pull say who wants them for missing objects.
This patch updates pull.c, the engine that decides which objects are
needed, given a commit to traverse from, to report which commit was
calling for the object that cannot be retrieved from the remote side.
This complements git-fsck-cache in that it checks the consistency of
the remote repository for reachability.
----------------------------------------------------------------
^ permalink raw reply
* Re: How to get bash to shut up about SIGPIPE?
From: Herbert Xu @ 2005-05-04 8:26 UTC (permalink / raw)
To: Linus Torvalds; +Cc: dwheeler, pj, git
In-Reply-To: <Pine.LNX.4.58.0505031947530.26698@ppc970.osdl.org>
Linus Torvalds <torvalds@osdl.org> wrote:
>
> But putting the traps _inside_ the loops seems to help. So something like
> the appended at least makes it somewhat useful
It's not the loop that's important, but the subshell where the
signal is caught. Every time you enter a subshell all traps are
reset. Since anything that comes (before or) after a pipe symbol
goes into a subshell...
> Index: cg-log
> ===================================================================
> --- aa6233be6d1b8bf42797c409a7c23b50593afc99/cg-log (mode:100755 sha1:aa2abf370753117a350818dbc91991b14d30ec6b)
> +++ uncommitted/cg-log (mode:100755)
> @@ -47,10 +47,12 @@
> fi
>
> $revls | $revsort | while read time commit parents; do
> + trap "exit 1" SIGPIPE
> [ "$revfmt" = "git-rev-list" ] && commit="$time"
> echo $colheader""commit ${commit%:*} $coldefault;
> git-cat-file commit $commit | \
> while read key rest; do
> + trap "exit 1" SIGPIPE
> case "$key" in
> "author"|"committer")
> if [ "$key" = "author" ]; then
You need it in both places because the signal may be received
in either place, depending on whether it's the echo or something
inside the loop that dies while writing output.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: 'read-tree -m head' vs 'read-tree head'
From: Thomas Glanzmann @ 2005-05-04 7:01 UTC (permalink / raw)
To: GIT
In-Reply-To: <7vr7gnpjkq.fsf@assigned-by-dhcp.cox.net>
Hello Junio,
> Normally "read-tree -m" is the preferred form from
> performance point of view, especially on a large project.
> The only case you need to use "read-tree" without -m is when
> the cache contains conflicting merge results and you want to
> start from scratch.
thanks for the bottom line. That explains why Linus uses it in his
git-pull-script.
Thomas
^ permalink raw reply
* Re: 'read-tree -m head' vs 'read-tree head'
From: Junio C Hamano @ 2005-05-04 6:54 UTC (permalink / raw)
To: Petr Baudis; +Cc: Thomas Glanzmann, GIT
In-Reply-To: <20050503213444.GD15995@pasky.ji.cz>
>>>>> "PB" == Petr Baudis <pasky@ucw.cz> writes:
PB> Dear diary, on Tue, May 03, 2005 at 09:13:40PM CEST, I got a letter
PB> where Junio C Hamano <junkio@cox.net> told me that...
>> That said, I've been wondering ... if there
>> is a valid use case where you would want to use it without "-m"
>> because "-m" does something wrong...
PB> -m fails when your cache file is missing/corrupted. Not that it cannot
PB> be fixed, just remember to fix it if you are going to do what you
PB> described.
Missing case seems to be handled fine. It spits out a message
from error() on a corrupt cache, but does not choke on it and
creates a valid cache in the end.
I just found out why we *do* need non -m version. The -m version
refuses to work on an unmerged cache, which is the right thing
to do. If you try a merge, and if you do not like the result,
you can throw the index away and start from scratch with non -m
version. Of course "rm .git/index" would work just fine as
well.
Another reason we may want to avoid -m version is that it takes
a long time to read the cache on a large project, but I think a
read-tree almost always is followed by a checkout-cache in a
typical use case, and -m version significantly improves the
performance of the latter.
read-tree -m $other; checkout-cache -f -a
read-tree $other; checkout-cache -f -a
A not so scientific benchmark of reading one commit in a work
tree based on a commit several steps away (this is linux-2.6
tree so the index file is 1.6MB), run on my machine with a slow
disk, does the former in 12 seconds and the latter in about 18
seconds. Comparison between the following is more drastic.
read-tree -m $other; update-cache --refresh
read-tree $other; update-cache --refresh
The version with -m wins by factor of ten.
So Thomas, here is my conclusion:
Normally "read-tree -m" is the preferred form from
performance point of view, especially on a large project.
The only case you need to use "read-tree" without -m is when
the cache contains conflicting merge results and you want to
start from scratch.
^ permalink raw reply
* [PATCH] Add git-relink-script, a tool to hardlink two existing repositories.
From: Ryan Anderson @ 2005-05-04 5:54 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Petr Baudis, git
In-Reply-To: <Pine.LNX.4.58.0504291311320.18901@ppc970.osdl.org>
Add git-relink-script, which will find common objects in two git
repositories and replace one copy with a hardlink.
Signed-Off-By: Ryan Anderson <ryan@michonline.com>
---
commit a3bcc763d71bdb91a3b48e9105fbaa5e79abb807
tree 2553e2d8befbe0cda3e413616fd4cc7bf04157ad
parent a31c6d022e2435a514fcc8ca57f9995c4376a986
author Ryan Anderson <ryan@mythryan2.(none)> 1115185675 -0400
committer Ryan Anderson <ryan@michonline.com> 1115185675 -0400
Index: Makefile
===================================================================
--- 51a882a2dc62e0d3cdc79e0badc61559fb723481/Makefile (mode:100644 sha1:99b4753d34879842b972da9b68694c9d0485f216)
+++ 2553e2d8befbe0cda3e413616fd4cc7bf04157ad/Makefile (mode:100644 sha1:a99665e252a2342caa84238e886a80a5f27ac3c8)
@@ -13,7 +13,7 @@
AR=ar
SCRIPTS=git-apply-patch-script git-merge-one-file-script git-prune-script \
- git-pull-script git-tag-script
+ git-pull-script git-tag-script git-relink-script
PROG= git-update-cache git-diff-files git-init-db git-write-tree \
git-read-tree git-commit-tree git-cat-file git-fsck-cache \
Index: git-relink-script
===================================================================
--- /dev/null (tree:51a882a2dc62e0d3cdc79e0badc61559fb723481)
+++ 2553e2d8befbe0cda3e413616fd4cc7bf04157ad/git-relink-script (mode:100644 sha1:78c954edcc370d8be951c856bfbfd38975d08348)
@@ -0,0 +1,115 @@
+#!/usr/bin/env perl
+# Copyright 2005, Ryan Anderson <ryan@michonline.com>
+# Distribution permitted under the GPL v2, as distributed
+# by the Free Software Foundation.
+# Later versions of the GPL at the discretion of Linus Torvalds
+#
+# Scan two git object-trees, and hardlink any common objects between them.
+
+use 5.006;
+use strict;
+use warnings;
+
+sub get_canonical_form($);
+sub do_scan_directory($$$);
+sub compare_two_files($$);
+
+# stats
+my $linked = 0;
+my $already = 0;
+
+my ($dir1, $dir2) = @ARGV;
+
+if (!defined $dir1 || !defined $dir2) {
+ print("Usage: $0 <dir1> <dir2>\nBoth dir1 and dir2 should contain a .git/objects/ subdirectory.\n");
+ exit(1);
+}
+
+$dir1 = get_canonical_form($dir1);
+$dir2 = get_canonical_form($dir2);
+
+printf("Searching '%s' and '%s' for common objects and hardlinking them...\n",$dir1,$dir2);
+
+opendir(D,$dir1 . "objects/")
+ or die "Failed to open $dir1/objects/ : $!";
+
+my @hashdirs = grep !/^\.{1,2}$/, readdir(D);
+foreach my $hashdir (@hashdirs) {
+ do_scan_directory($dir1, $hashdir, $dir2);
+}
+
+printf("Linked %d files, %d were already linked.\n",$linked, $already);
+
+
+sub do_scan_directory($$$) {
+ my ($srcdir, $subdir, $dstdir) = @_;
+
+ my $sfulldir = sprintf("%sobjects/%s/",$srcdir,$subdir);
+ my $dfulldir = sprintf("%sobjects/%s/",$dstdir,$subdir);
+
+ opendir(S,$sfulldir)
+ or die "Failed to opendir $sfulldir: $!";
+
+ foreach my $file (grep(!/\.{1,2}$/, readdir(S))) {
+ my $sfilename = $sfulldir . $file;
+ my $dfilename = $dfulldir . $file;
+
+ compare_two_files($sfilename,$dfilename);
+
+ }
+ closedir(S);
+}
+
+sub compare_two_files($$) {
+ my ($sfilename, $dfilename) = @_;
+
+ # Perl's stat returns relevant information as follows:
+ # 0 = dev number
+ # 1 = inode number
+ # 7 = size
+ my @sstatinfo = stat($sfilename);
+ my @dstatinfo = stat($dfilename);
+
+ if (@sstatinfo == 0 && @dstatinfo == 0) {
+ die sprintf("Stat of both %s and %s failed: %s\n",$sfilename, $dfilename, $!);
+
+ } elsif (@dstatinfo == 0) {
+ return;
+ }
+
+ if ( ($sstatinfo[0] == $dstatinfo[0]) &&
+ ($sstatinfo[1] != $dstatinfo[1])) {
+ if ($sstatinfo[7] == $dstatinfo[7]) {
+ unlink($dfilename)
+ or die "Unlink of $dfilename failed: $!\n";
+
+ link($sfilename,$dfilename)
+ or die "Failed to link $sfilename to $dfilename: $!\n" .
+ "Git Repository containing $dfilename is probably corrupted, please copy '$sfilename' to '$dfilename' to fix.\n";
+
+ $linked++;
+
+ } else {
+ die sprintf("ERROR: File sizes are not the same, cannot relink %s to %s.\n",
+ $sfilename, $dfilename);
+ }
+
+ } elsif ( ($sstatinfo[0] == $dstatinfo[0]) &&
+ ($sstatinfo[1] == $dstatinfo[1])) {
+ $already++;
+ }
+}
+
+sub get_canonical_form($) {
+ my $dir = shift;
+ my $original = $dir;
+
+ die "$dir is not a directory." unless -d $dir;
+
+ $dir .= "/" unless $dir =~ m#/$#;
+ $dir .= ".git/" unless $dir =~ m#\.git/$#;
+
+ die "$original does not have a .git/ subdirectory.\n" unless -d $dir;
+
+ return $dir;
+}
--
Ryan Anderson
sometimes Pug Majere
^ permalink raw reply
* Re: [PATCH] cogito: Add cg-undo command
From: Matt Porter @ 2005-05-04 5:40 UTC (permalink / raw)
To: Petr Baudis; +Cc: git
In-Reply-To: <20050503213204.GC15995@pasky.ji.cz>
On Tue, May 03, 2005 at 11:32:04PM +0200, Petr Baudis wrote:
> Dear diary, on Tue, May 03, 2005 at 07:06:25PM CEST, I got a letter
> where Matt Porter <mporter@kernel.crashing.org> told me that...
> > + cg-undo [COMMIT_ID]
>
> It doesn't seem very optional now.
Yep, changed.
> > +PARENT=`git-cat-file commit $1 | grep parent | cut -f 2 -d " "`
>
> What's wrong with parent-id?
Missed it...changed to use parent-id.
> > +echo "Undo from $1 to current HEAD"
> > +echo "Reset HEAD to $PARENT"
>
> You talk way too much, I think. I'd just do
>
> echo "Rewinding $HEAD -> $PARENT" >&2
Done.
> > +git-update-cache --refresh
>
> You really don't want to do this if the tree has any local
> modifications.
Ugh, yes. I added a check for this.
> Please make sure the commit you are rewinding to is an ancestor of your
> current HEAD (there should be something like separate cg-branch-rm for
> killing enemy branches).
Added a check for this as well as verifying we have an argument.
Signed-off-by: Matt Porter <mporter@kernel.crashing.org>
--- aa6233be6d1b8bf42797c409a7c23b50593afc99/Makefile (mode:100644 sha1:6ae0afa0208a8f755d383281a6d049a4ef90fe63)
+++ ce72371d991b15a0ca8db7c2332d215b59b909af/Makefile (mode:100644 sha1:6c282aeebe86ecee9e634481b3d51fd53a582791)
@@ -47,7 +47,7 @@
cg-add cg-admin-lsobj cg-cancel cg-clone cg-commit cg-diff \
cg-export cg-help cg-init cg-log cg-ls cg-merge cg-mkpatch \
cg-patch cg-pull cg-branch-add cg-branch-ls cg-rm cg-seek cg-status \
- cg-tag cg-tag-ls cg-update cg-Xlib
+ cg-tag cg-tag-ls cg-undo cg-update cg-Xlib
COMMON= read-cache.o
Index: cg-help
===================================================================
--- aa6233be6d1b8bf42797c409a7c23b50593afc99/cg-help (mode:100755 sha1:1f5d2d79b67490d44ce0f575ff9a4b80134ea47f)
+++ ce72371d991b15a0ca8db7c2332d215b59b909af/cg-help (mode:100755 sha1:1b75114ee90d2b3b9786fc4f14bf179feef54f87)
@@ -43,6 +43,7 @@
cg-status
cg-tag TNAME [COMMIT_ID]
cg-tag-ls
+ cg-undo COMMIT_ID
cg-update [BNAME]
cg-version
Index: cg-undo
===================================================================
--- /dev/null (tree:aa6233be6d1b8bf42797c409a7c23b50593afc99)
+++ ce72371d991b15a0ca8db7c2332d215b59b909af/cg-undo (mode:100644 sha1:7b1cb04c79731b137d88ebd05ee41553af7246d2)
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+#
+# Undo a commit or a series of commits
+# Copyright (C) Matt Porter, 2005
+#
+# Takes a commit ID which is the earliest commit to be
+# removed from the repository.
+
+. cg-Xlib
+
+[ "$1" ] || die "usage: cg-undo COMMIT_ID"
+
+HEAD=$(commit-id) || exit 1;
+PARENT=$(parent-id "$1") || exit 1;
+
+git-rev-list $HEAD | grep -q $1 || {
+ echo >&2 "$1: not an ancestor of HEAD"
+ exit 1
+}
+[ "$(git-diff-files -s)" ] && git-update-cache --refresh
+if [ "$(git-diff-files -s)" ] || [ "$(git-diff-cache $(tree-id))" ]; then
+ die "Undo blocked: local changes"
+fi
+
+echo "Rewinding HEAD -> $PARENT" >&2
+echo "$PARENT" > .git/HEAD
+git-read-tree -m "$PARENT" || {
+ echo >&2 "$PARENT: bad commit"
+ exit 1
+}
+git-checkout-cache -f -a
+git-update-cache --refresh
^ permalink raw reply
* [PATCH] Careful object pulling
From: Daniel Barkalow @ 2005-05-04 4:07 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Git Mailing List
In-Reply-To: <Pine.LNX.4.58.0505031204030.26698@ppc970.osdl.org>
This splits out the careful methods for writing a file and placing it in
the correct location from the function to write a buffer to the file and
makes the various functions used by git-*-pull programs use those
functions to write their files.
Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Index: cache.h
===================================================================
--- 51a882a2dc62e0d3cdc79e0badc61559fb723481/cache.h (mode:100644 sha1:8dd812827604d510038f5d93e3718c43f9d12c30)
+++ 4e31436bacfff09ce673665a1061b41e37ffd661/cache.h (mode:100644 sha1:0d7411c3b86a899cee45627997f4bb7ba0df2ea7)
@@ -134,6 +134,12 @@
extern void * read_sha1_file(const unsigned char *sha1, char *type, unsigned long *size);
extern int write_sha1_file(char *buf, unsigned long len, const char *type, unsigned char *return_sha1);
+/* Open a file for writing a sha1 file. */
+extern int open_sha1_tmpfile(char tmpfile[PATH_MAX]);
+
+/* Put a sha1 file in the correct place. */
+extern int place_sha1_file(char tmpfile[PATH_MAX], const unsigned char *sha1);
+
extern int check_sha1_signature(unsigned char *sha1, void *buf, unsigned long size, const char *type);
/* Read a tree into the cache */
Index: http-pull.c
===================================================================
--- 51a882a2dc62e0d3cdc79e0badc61559fb723481/http-pull.c (mode:100644 sha1:f693aba61b4dcb4b738faf1335ec74aa1545e45d)
+++ 4e31436bacfff09ce673665a1061b41e37ffd661/http-pull.c (mode:100644 sha1:793d8b9e125c1f164f774e2888d26beaa75e1df0)
@@ -52,8 +52,9 @@
char real_sha1[20];
char *url;
char *posn;
+ char tmpname[PATH_MAX];
- local = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);
+ local = open_sha1_tmpfile(tmpname);
if (local < 0)
return error("Couldn't open %s\n", filename);
@@ -88,15 +89,14 @@
inflateEnd(&stream);
SHA1_Final(real_sha1, &c);
if (zret != Z_STREAM_END) {
- unlink(filename);
+ unlink(tmpname);
return error("File %s (%s) corrupt\n", hex, url);
}
if (memcmp(sha1, real_sha1, 20)) {
- unlink(filename);
+ unlink(tmpname);
return error("File %s has bad hash\n", hex);
}
-
- return 0;
+ return place_sha1_file(tmpname, sha1);
}
int main(int argc, char **argv)
Index: sha1_file.c
===================================================================
--- 51a882a2dc62e0d3cdc79e0badc61559fb723481/sha1_file.c (mode:100644 sha1:e6ce455ae90bd430f2128f454bdb6e0575412486)
+++ 4e31436bacfff09ce673665a1061b41e37ffd661/sha1_file.c (mode:100644 sha1:85daa0b0045c3f19e697d1a7aa8ab15ff54eab99)
@@ -276,6 +276,55 @@
}
}
+int open_sha1_tmpfile(char tmpfile[PATH_MAX])
+{
+ int fd;
+
+ snprintf(tmpfile, sizeof(tmpfile), "%s/obj_XXXXXX", get_object_directory());
+ fd = mkstemp(tmpfile);
+ if (fd < 0) {
+ fprintf(stderr,
+ "unable to create temporary sha1 filename %s: %s",
+ tmpfile, strerror(errno));
+ return -1;
+ }
+ return fd;
+}
+
+int place_sha1_file(char tmpfile[PATH_MAX], const unsigned char *sha1)
+{
+ char *filename = sha1_file_name(sha1);
+ int ret;
+
+ ret = link(tmpfile, filename);
+ if (ret < 0) {
+ ret = errno;
+
+ /*
+ * Coda hack - coda doesn't like cross-directory links,
+ * so we fall back to a rename, which will mean that it
+ * won't be able to check collisions, but that's not a
+ * big deal.
+ *
+ * When this succeeds, we just return 0. We have nothing
+ * left to unlink.
+ */
+ if (ret == EXDEV && !rename(tmpfile, filename))
+ return 0;
+ }
+ unlink(tmpfile);
+ if (ret) {
+ if (ret != EEXIST) {
+ fprintf(stderr,
+ "unable to write sha1 filename %s: %s",
+ filename, strerror(ret));
+ return -1;
+ }
+ /* FIXME!!! Collision check here ? */
+ }
+ return 0;
+}
+
int write_sha1_file(char *buf, unsigned long len, const char *type, unsigned char *returnsha1)
{
int size;
@@ -286,7 +335,7 @@
char *filename;
static char tmpfile[PATH_MAX];
char hdr[50];
- int fd, hdrlen, ret;
+ int fd, hdrlen;
/* Generate the header */
hdrlen = sprintf(hdr, "%s %lu", type, len)+1;
@@ -316,12 +365,9 @@
return -1;
}
- snprintf(tmpfile, sizeof(tmpfile), "%s/obj_XXXXXX", get_object_directory());
- fd = mkstemp(tmpfile);
- if (fd < 0) {
- fprintf(stderr, "unable to create temporary sha1 filename %s: %s", tmpfile, strerror(errno));
- return -1;
- }
+ fd = open_sha1_tmpfile(tmpfile);
+ if (fd < 0)
+ return fd;
/* Set it up */
memset(&stream, 0, sizeof(stream));
@@ -349,53 +395,28 @@
if (write(fd, compressed, size) != size)
die("unable to write file");
+
fchmod(fd, 0444);
close(fd);
- ret = link(tmpfile, filename);
- if (ret < 0) {
- ret = errno;
-
- /*
- * Coda hack - coda doesn't like cross-directory links,
- * so we fall back to a rename, which will mean that it
- * won't be able to check collisions, but that's not a
- * big deal.
- *
- * When this succeeds, we just return 0. We have nothing
- * left to unlink.
- */
- if (ret == EXDEV && !rename(tmpfile, filename))
- return 0;
- }
- unlink(tmpfile);
- if (ret) {
- if (ret != EEXIST) {
- fprintf(stderr, "unable to write sha1 filename %s: %s", filename, strerror(ret));
- return -1;
- }
- /* FIXME!!! Collision check here ? */
- }
-
- return 0;
+ return place_sha1_file(tmpfile, sha1);
}
int write_sha1_from_fd(const unsigned char *sha1, int fd)
{
- char *filename = sha1_file_name(sha1);
-
int local;
z_stream stream;
unsigned char real_sha1[20];
char buf[4096];
char discard[4096];
+ char tmpname[PATH_MAX];
int ret;
SHA_CTX c;
- local = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);
+ local = open_sha1_tmpfile(tmpname);
if (local < 0)
- return error("Couldn't open %s\n", filename);
+ return -1;
memset(&stream, 0, sizeof(stream));
@@ -408,7 +429,7 @@
size = read(fd, buf, 4096);
if (size <= 0) {
close(local);
- unlink(filename);
+ unlink(tmpname);
if (!size)
return error("Connection closed?");
perror("Reading from connection");
@@ -431,15 +452,15 @@
close(local);
SHA1_Final(real_sha1, &c);
if (ret != Z_STREAM_END) {
- unlink(filename);
+ unlink(tmpname);
return error("File %s corrupted", sha1_to_hex(sha1));
}
if (memcmp(sha1, real_sha1, 20)) {
- unlink(filename);
+ unlink(tmpname);
return error("File %s has bad hash\n", sha1_to_hex(sha1));
}
-
- return 0;
+
+ return place_sha1_file(tmpname, sha1);
}
int has_sha1_file(const unsigned char *sha1)
^ permalink raw reply
* Mercurial v0.4c
From: Matt Mackall @ 2005-05-04 2:58 UTC (permalink / raw)
To: linux-kernel, git, Linus Torvalds
A new version of Mercurial is available at:
http://selenic.com/mercurial/
This version is officially self-hosting, now that I've added the final
planned changed to the metadata. To pull the repo, do:
hg init
hg merge http://selenic.com/hg
This version fixes numerous reported bugs, adds a "verify" command to
check the repository integrity, transaction handling, and some minor
speed improvements.
--
Mathematics is the supreme nostalgia of our time.
^ permalink raw reply
* Re: How to get bash to shut up about SIGPIPE?
From: Linus Torvalds @ 2005-05-04 2:50 UTC (permalink / raw)
To: David A. Wheeler; +Cc: Paul Jackson, Git Mailing List
In-Reply-To: <427833AE.6030505@dwheeler.com>
On Tue, 3 May 2005, David A. Wheeler wrote:
>
> I wonder, does a top-level trap work? E.g.:
> trap "" SIGPIPE
No.
But putting the traps _inside_ the loops seems to help. So something like
the appended at least makes it somewhat useful
And yes, you need them at both levels, it appears. Or maybe that just
changes the timing enough. Whatever.
Linus
----
Index: cg-log
===================================================================
--- aa6233be6d1b8bf42797c409a7c23b50593afc99/cg-log (mode:100755 sha1:aa2abf370753117a350818dbc91991b14d30ec6b)
+++ uncommitted/cg-log (mode:100755)
@@ -47,10 +47,12 @@
fi
$revls | $revsort | while read time commit parents; do
+ trap "exit 1" SIGPIPE
[ "$revfmt" = "git-rev-list" ] && commit="$time"
echo $colheader""commit ${commit%:*} $coldefault;
git-cat-file commit $commit | \
while read key rest; do
+ trap "exit 1" SIGPIPE
case "$key" in
"author"|"committer")
if [ "$key" = "author" ]; then
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox