* Re: Index/hash order
From: Linus Torvalds @ 2005-04-13 22:11 UTC (permalink / raw)
To: Florian Weimer; +Cc: H. Peter Anvin, Ingo Molnar, git
In-Reply-To: <87aco2gxu2.fsf@deneb.enyo.de>
On Wed, 13 Apr 2005, Florian Weimer wrote:
>
> I've run a few tests, just to get a few numbers of the overhead
> involved. I used the last ~8,000 changesets from the BKCVS kernel
> repository. With cold cache, a checkout from cold cache takes about
> 250 seconds on my laptop. I don't have git numbers, but a mere copy
> of the kernel tree needs 40 seconds.
I will bet you that a git checkout is _faster_ than a kernel source tree
copy. The time will be dominated by the IO costs (in particular the read
costs), and the IO costs are lower thanks to compression. So I think that
the cold-cache case will beat your 40 seconds by a clear margin. It
generally compresses to half the size, so 20 seconds is not impossible
(although seek costs would tend to stay constant, so I'd expect it to be
somewhere in between the two).
> For the hot-cache case, the difference is 140 seconds vs. 2.5 seconds
> (or 6 seconds with decompression).
>
> Uh-oh. I wouldn't have imaged the difference would be *that*
> dramatic. The file system layer is *fast*.
Did I mention that I designed git for speed?
Yes. The whole damn design is really about performance, distribution, and
built-in integrity checking.
> On the storage front, we have 220 MB for the skip deltas vs. 106 MB
> for pure deltas-to-previous vs. 1.1 GB for uncompressed files
> (directories are always delta-compressed, so to speak[1]).
That's actually pretty encouraging. Your 1.1GB number implies to me that a
compressed file setup should be about half that, which in turn says that
the cost of full-file is not at all outrageous. Sure, it's 2-3 times
larger than your skip deltas, but considering that the performance is
about fifty times faster (and I can do distributed stuff without any
locking synchronization and you can't), that's a tradeoff I'm more than
happy with.
Or maybe I misunderstood what you were comparing?
Of course, the numbers will all depend on how the history looks etc, so
this is all pretty much just guidelines.
Linus
^ permalink raw reply
* [PATCH] Reorganize common code
From: Daniel Barkalow @ 2005-04-13 22:00 UTC (permalink / raw)
To: Petr Baudis; +Cc: git
In-Reply-To: <Pine.LNX.4.21.0504131752410.30848-100000@iabervon.org>
This splits read-cache.c into util.c, cache.c, and objects.c, based on
what the code is used for; similarly, cache.h becomes util.h, cache.h, and
objects.h. For simplicity, cache.h includes the other two to give the
previous overall behavior.
Signed-Off-By: Daniel Barkalow <barkalow@iabervon.org>
Index: Makefile
===================================================================
--- 783978eb212c1402ba2612bb32626e93af78b72d/Makefile (mode:100644 sha1:2b626ec05b2e29016d26b5cb7b4f82ec5c4e79d6)
+++ e8194c62bfc68725972a6847fa2c6d529ca64137/Makefile (mode:100644 sha1:0b263fa693cb2addb2516a8f7af2ea05b95d3c8c)
@@ -20,7 +20,7 @@
gitmerge.sh gitpull.sh gitrm.sh gittag.sh gittrack.sh gitexport.sh \
gitapply.sh
-COMMON= read-cache.o
+COMMON= cache.o objects.o util.o
GEN_SCRIPT= gitversion.sh
@@ -36,7 +36,7 @@
$(PROG):%: %.o $(COMMON)
$(CC) $(CFLAGS) -o $@ $^ $(LIBS)
-read-cache.o: cache.h
+cache.o: cache.h
show-diff.o: cache.h
gitversion.sh: $(VERSION)
Index: cache.c
===================================================================
--- /dev/null (tree:783978eb212c1402ba2612bb32626e93af78b72d)
+++ e8194c62bfc68725972a6847fa2c6d529ca64137/cache.c (mode:100644 sha1:fe61eae561e8ebaab9d9fe6d8f146ed7353b6a1c)
@@ -0,0 +1,224 @@
+/*
+ * GIT - The information manager from hell
+ *
+ * Copyright (C) Linus Torvalds, 2005
+ */
+#include <stdarg.h>
+#include "cache.h"
+
+const char *sha1_file_directory = NULL;
+struct cache_entry **active_cache = NULL;
+unsigned int active_nr = 0, active_alloc = 0;
+
+int cache_match_stat(struct cache_entry *ce, struct stat *st)
+{
+ unsigned int changed = 0;
+
+ /* nsec seems unreliable - not all filesystems support it, so
+ * as long as it is in the inode cache you get right nsec
+ * but after it gets flushed, you get zero nsec. */
+ if (ce->mtime.sec != (unsigned int)st->st_mtim.tv_sec
+#ifdef NSEC
+ || ce->mtime.nsec != (unsigned int)st->st_mtim.tv_nsec
+#endif
+ )
+ changed |= MTIME_CHANGED;
+ if (ce->ctime.sec != (unsigned int)st->st_ctim.tv_sec
+#ifdef NSEC
+ || ce->ctime.nsec != (unsigned int)st->st_ctim.tv_nsec
+#endif
+ )
+ changed |= CTIME_CHANGED;
+ if (ce->st_uid != (unsigned int)st->st_uid ||
+ ce->st_gid != (unsigned int)st->st_gid)
+ changed |= OWNER_CHANGED;
+ if (ce->st_mode != (unsigned int)st->st_mode)
+ changed |= MODE_CHANGED;
+ if (ce->st_dev != (unsigned int)st->st_dev ||
+ ce->st_ino != (unsigned int)st->st_ino)
+ changed |= INODE_CHANGED;
+ if (ce->st_size != (unsigned int)st->st_size)
+ changed |= DATA_CHANGED;
+ return changed;
+}
+
+int cache_name_compare(const char *name1, int len1, const char *name2, int len2)
+{
+ int len = len1 < len2 ? len1 : len2;
+ int cmp;
+
+ cmp = memcmp(name1, name2, len);
+ if (cmp)
+ return cmp;
+ if (len1 < len2)
+ return -1;
+ if (len1 > len2)
+ return 1;
+ return 0;
+}
+
+int cache_name_pos(const char *name, int namelen)
+{
+ int first, last;
+
+ first = 0;
+ last = active_nr;
+ while (last > first) {
+ int next = (last + first) >> 1;
+ struct cache_entry *ce = active_cache[next];
+ int cmp = cache_name_compare(name, namelen, ce->name, ce->namelen);
+ if (!cmp)
+ return next;
+ if (cmp < 0) {
+ last = next;
+ continue;
+ }
+ first = next+1;
+ }
+ return -first-1;
+}
+
+int remove_file_from_cache(char *path)
+{
+ int pos = cache_name_pos(path, strlen(path));
+ if (pos >= 0) {
+ active_nr--;
+ if (pos < active_nr)
+ memmove(active_cache + pos, active_cache + pos + 1, (active_nr - pos) * sizeof(struct cache_entry *));
+ }
+ return 0;
+}
+
+int add_cache_entry(struct cache_entry *ce, int ok_to_add)
+{
+ int pos;
+
+ pos = cache_name_pos(ce->name, ce->namelen);
+
+ /* existing match? Just replace it */
+ if (pos >= 0) {
+ active_cache[pos] = ce;
+ return 0;
+ }
+ pos = -pos-1;
+
+ if (!ok_to_add)
+ return -1;
+
+ /* Make sure the array is big enough .. */
+ if (active_nr == active_alloc) {
+ active_alloc = alloc_nr(active_alloc);
+ active_cache = realloc(active_cache, active_alloc * sizeof(struct cache_entry *));
+ }
+
+ /* Add it in.. */
+ active_nr++;
+ if (active_nr > pos)
+ memmove(active_cache + pos + 1, active_cache + pos, (active_nr - pos - 1) * sizeof(ce));
+ active_cache[pos] = ce;
+ return 0;
+}
+
+static int verify_hdr(struct cache_header *hdr, unsigned long size)
+{
+ SHA_CTX c;
+ unsigned char sha1[20];
+
+ if (hdr->signature != CACHE_SIGNATURE)
+ return error("bad signature");
+ if (hdr->version != 1)
+ return error("bad version");
+ SHA1_Init(&c);
+ SHA1_Update(&c, hdr, offsetof(struct cache_header, sha1));
+ SHA1_Update(&c, hdr+1, size - sizeof(*hdr));
+ SHA1_Final(sha1, &c);
+ if (memcmp(sha1, hdr->sha1, 20))
+ return error("bad header sha1");
+ return 0;
+}
+
+int read_cache(void)
+{
+ int fd, i;
+ struct stat st;
+ unsigned long size, offset;
+ void *map;
+ struct cache_header *hdr;
+
+ errno = EBUSY;
+ if (active_cache)
+ return error("more than one cachefile");
+ errno = ENOENT;
+ sha1_file_directory = getenv(DB_ENVIRONMENT);
+ if (!sha1_file_directory)
+ sha1_file_directory = DEFAULT_DB_ENVIRONMENT;
+ if (access(sha1_file_directory, X_OK) < 0)
+ return error("no access to SHA1 file directory");
+ fd = open(".git/index", O_RDONLY);
+ if (fd < 0)
+ return (errno == ENOENT) ? 0 : error("open failed");
+
+ size = 0; // avoid gcc warning
+ map = (void *)-1;
+ if (!fstat(fd, &st)) {
+ size = st.st_size;
+ errno = EINVAL;
+ if (size >= sizeof(struct cache_header))
+ map = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);
+ }
+ close(fd);
+ if (-1 == (int)(long)map)
+ return error("mmap failed");
+
+ hdr = map;
+ if (verify_hdr(hdr, size) < 0)
+ goto unmap;
+
+ active_nr = hdr->entries;
+ active_alloc = alloc_nr(active_nr);
+ active_cache = calloc(active_alloc, sizeof(struct cache_entry *));
+
+ offset = sizeof(*hdr);
+ for (i = 0; i < hdr->entries; i++) {
+ struct cache_entry *ce = map + offset;
+ offset = offset + ce_size(ce);
+ active_cache[i] = ce;
+ }
+ return active_nr;
+
+unmap:
+ munmap(map, size);
+ errno = EINVAL;
+ return error("verify header failed");
+}
+
+int write_cache(int newfd, struct cache_entry **cache, int entries)
+{
+ SHA_CTX c;
+ struct cache_header hdr;
+ int i;
+
+ hdr.signature = CACHE_SIGNATURE;
+ hdr.version = 1;
+ hdr.entries = entries;
+
+ SHA1_Init(&c);
+ SHA1_Update(&c, &hdr, offsetof(struct cache_header, sha1));
+ for (i = 0; i < entries; i++) {
+ struct cache_entry *ce = cache[i];
+ int size = ce_size(ce);
+ SHA1_Update(&c, ce, size);
+ }
+ SHA1_Final(hdr.sha1, &c);
+
+ if (write(newfd, &hdr, sizeof(hdr)) != sizeof(hdr))
+ return -1;
+
+ for (i = 0; i < entries; i++) {
+ struct cache_entry *ce = cache[i];
+ int size = ce_size(ce);
+ if (write(newfd, ce, size) != size)
+ return -1;
+ }
+ return 0;
+}
Index: cache.h
===================================================================
--- 783978eb212c1402ba2612bb32626e93af78b72d/cache.h (mode:100644 sha1:d3e9a21b7d9a2ac32abacf5cc40ee1a4d83f9fe8)
+++ e8194c62bfc68725972a6847fa2c6d529ca64137/cache.h (mode:100644 sha1:e3302b454e5d55d4d7bd46bbfe870cb34733ce46)
@@ -14,6 +14,9 @@
#include <openssl/sha.h>
#include <zlib.h>
+#include "util.h"
+#include "objects.h"
+
/*
* Basic data structures for the directory cache
*
@@ -59,13 +62,9 @@
char name[0];
};
-const char *sha1_file_directory;
struct cache_entry **active_cache;
unsigned int active_nr, active_alloc;
-#define DB_ENVIRONMENT "SHA1_FILE_DIRECTORY"
-#define DEFAULT_DB_ENVIRONMENT ".git/objects"
-
#define cache_entry_size(len) ((offsetof(struct cache_entry,name) + (len) + 8) & ~7)
#define ce_size(ce) cache_entry_size((ce)->namelen)
@@ -74,7 +73,10 @@
/* Initialize and use the cache information */
extern int read_cache(void);
extern int write_cache(int newfd, struct cache_entry **cache, int entries);
+
+/* Returns the location of the given name in the cache. */
extern int cache_name_pos(const char *name, int namelen);
+
extern int add_cache_entry(struct cache_entry *ce, int ok_to_add);
extern int remove_file_from_cache(char *path);
extern int cache_match_stat(struct cache_entry *ce, struct stat *st);
@@ -86,28 +88,7 @@
#define INODE_CHANGED 0x0010
#define DATA_CHANGED 0x0020
-/* Return a statically allocated filename matching the sha1 signature */
-extern char *sha1_file_name(const unsigned char *sha1);
-
-/* Write a memory buffer out to the sha file */
-extern int write_sha1_buffer(const unsigned char *sha1, void *buf, unsigned int size);
-
-/* Read and unpack a sha1 file into memory, write memory to a sha1 file */
-extern void * map_sha1_file(const unsigned char *sha1, unsigned long *size);
-extern void * unpack_sha1_file(void *map, unsigned long mapsize, char *type, unsigned long *size);
-extern void * read_sha1_file(const unsigned char *sha1, char *type, unsigned long *size);
-extern int write_sha1_file(char *buf, unsigned len, unsigned char *return_sha1);
-extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned long size);
-
-/* Convert to/from hex/sha1 representation */
-extern int get_sha1_hex(const char *hex, unsigned char *sha1);
-extern char *sha1_to_hex(const unsigned char *sha1); /* static buffer result! */
-
-/* General helper functions */
-extern void usage(const char *err);
-extern void die(const char *err, ...);
-extern int error(const char *err, ...);
-
-extern int cache_name_compare(const char *name1, int len1, const char *name2, int len2);
+extern int cache_name_compare(const char *name1, int len1,
+ const char *name2, int len2);
#endif /* CACHE_H */
Index: objects.c
===================================================================
--- /dev/null (tree:783978eb212c1402ba2612bb32626e93af78b72d)
+++ e8194c62bfc68725972a6847fa2c6d529ca64137/objects.c (mode:100644 sha1:ee0aed670a76e0af9e4150c93915dad53a198ba5)
@@ -0,0 +1,240 @@
+/*
+ * GIT - The information manager from hell
+ *
+ * Copyright (C) Linus Torvalds, 2005
+ */
+#include <stdarg.h>
+#include "cache.h"
+#include "objects.h"
+#include <openssl/sha.h>
+#include <stdio.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <stddef.h>
+#include <stdlib.h>
+#include <stdarg.h>
+#include <string.h>
+#include <errno.h>
+#include <sys/mman.h>
+#include <zlib.h>
+
+
+static unsigned hexval(char c)
+{
+ if (c >= '0' && c <= '9')
+ return c - '0';
+ if (c >= 'a' && c <= 'f')
+ return c - 'a' + 10;
+ if (c >= 'A' && c <= 'F')
+ return c - 'A' + 10;
+ return ~0;
+}
+
+int get_sha1_hex(const char *hex, unsigned char *sha1)
+{
+ int i;
+ for (i = 0; i < 20; i++) {
+ unsigned int val = (hexval(hex[0]) << 4) | hexval(hex[1]);
+ if (val & ~0xff)
+ return -1;
+ *sha1++ = val;
+ hex += 2;
+ }
+ return 0;
+}
+
+char * sha1_to_hex(const unsigned char *sha1)
+{
+ static char buffer[50];
+ static const char hex[] = "0123456789abcdef";
+ char *buf = buffer;
+ int i;
+
+ for (i = 0; i < 20; i++) {
+ unsigned int val = *sha1++;
+ *buf++ = hex[val >> 4];
+ *buf++ = hex[val & 0xf];
+ }
+ return buffer;
+}
+
+/*
+ * NOTE! This returns a statically allocated buffer, so you have to be
+ * careful about using it. Do a "strdup()" if you need to save the
+ * filename.
+ */
+char *sha1_file_name(const unsigned char *sha1)
+{
+ int i;
+ static char *name, *base;
+
+ if (!base) {
+ char *sha1_file_directory = getenv(DB_ENVIRONMENT) ? : DEFAULT_DB_ENVIRONMENT;
+ int len = strlen(sha1_file_directory);
+ base = malloc(len + 60);
+ memcpy(base, sha1_file_directory, len);
+ memset(base+len, 0, 60);
+ base[len] = '/';
+ base[len+3] = '/';
+ name = base + len + 1;
+ }
+ for (i = 0; i < 20; i++) {
+ static char hex[] = "0123456789abcdef";
+ unsigned int val = sha1[i];
+ char *pos = name + i*2 + (i > 0);
+ *pos++ = hex[val >> 4];
+ *pos = hex[val & 0xf];
+ }
+ return base;
+}
+
+int check_sha1_signature(const unsigned char *sha1, void *map, unsigned long size)
+{
+ unsigned char real_sha1[20];
+ SHA_CTX c;
+
+ SHA1_Init(&c);
+ SHA1_Update(&c, map, size);
+ SHA1_Final(real_sha1, &c);
+ return memcmp(sha1, real_sha1, 20) ? -1 : 0;
+}
+
+void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
+{
+ char *filename = sha1_file_name(sha1);
+ int fd = open(filename, O_RDONLY);
+ struct stat st;
+ void *map;
+
+ if (fd < 0) {
+ perror(filename);
+ return NULL;
+ }
+ if (fstat(fd, &st) < 0) {
+ close(fd);
+ return NULL;
+ }
+ map = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
+ close(fd);
+ if (-1 == (int)(long)map)
+ return NULL;
+ *size = st.st_size;
+ return map;
+}
+
+void * unpack_sha1_file(void *map, unsigned long mapsize, char *type, unsigned long *size)
+{
+ int ret, bytes;
+ z_stream stream;
+ char buffer[8192];
+ char *buf;
+
+ /* Get the data stream */
+ memset(&stream, 0, sizeof(stream));
+ stream.next_in = map;
+ stream.avail_in = mapsize;
+ stream.next_out = buffer;
+ stream.avail_out = sizeof(buffer);
+
+ inflateInit(&stream);
+ ret = inflate(&stream, 0);
+ if (sscanf(buffer, "%10s %lu", type, size) != 2)
+ return NULL;
+
+ bytes = strlen(buffer) + 1;
+ buf = malloc(*size);
+ if (!buf)
+ return NULL;
+
+ memcpy(buf, buffer + bytes, stream.total_out - bytes);
+ bytes = stream.total_out - bytes;
+ if (bytes < *size && ret == Z_OK) {
+ stream.next_out = buf + bytes;
+ stream.avail_out = *size - bytes;
+ while (inflate(&stream, Z_FINISH) == Z_OK)
+ /* nothing */;
+ }
+ inflateEnd(&stream);
+ return buf;
+}
+
+void * read_sha1_file(const unsigned char *sha1, char *type, unsigned long *size)
+{
+ unsigned long mapsize;
+ void *map, *buf;
+
+ map = map_sha1_file(sha1, &mapsize);
+ if (map) {
+ buf = unpack_sha1_file(map, mapsize, type, size);
+ munmap(map, mapsize);
+ return buf;
+ }
+ return NULL;
+}
+
+int write_sha1_file(char *buf, unsigned len, unsigned char *returnsha1)
+{
+ int size;
+ char *compressed;
+ z_stream stream;
+ unsigned char sha1[20];
+ SHA_CTX c;
+
+ /* Set it up */
+ memset(&stream, 0, sizeof(stream));
+ deflateInit(&stream, Z_BEST_COMPRESSION);
+ size = deflateBound(&stream, len);
+ compressed = malloc(size);
+
+ /* Compress it */
+ stream.next_in = buf;
+ stream.avail_in = len;
+ stream.next_out = compressed;
+ stream.avail_out = size;
+ while (deflate(&stream, Z_FINISH) == Z_OK)
+ /* nothing */;
+ deflateEnd(&stream);
+ size = stream.total_out;
+
+ /* Sha1.. */
+ SHA1_Init(&c);
+ SHA1_Update(&c, compressed, size);
+ SHA1_Final(sha1, &c);
+
+ if (write_sha1_buffer(sha1, compressed, size) < 0)
+ return -1;
+ if (returnsha1)
+ memcpy(returnsha1, sha1, 20);
+ return 0;
+}
+
+int write_sha1_buffer(const unsigned char *sha1, void *buf, unsigned int size)
+{
+ char *filename = sha1_file_name(sha1);
+ int fd;
+
+ fd = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);
+ if (fd < 0) {
+ void *map;
+
+ if (errno != EEXIST)
+ return -1;
+ /* #define NO_COLLISION_CHECK to possibly speed things up,
+ * but the speedup should be really negligible. */
+#ifndef NO_COLLISION_CHECK
+ fd = open(filename, O_RDONLY);
+ if (fd < 0)
+ return -1;
+ map = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);
+ if (map == MAP_FAILED)
+ return -1;
+ if (memcmp(buf, map, size))
+ return error("SHA1 collision detected!"
+ " This is bad, bad, BAD!\a\n");
+#endif
+ return 0;
+ }
+ write(fd, buf, size);
+ close(fd);
+ return 0;
+}
Index: objects.h
===================================================================
--- /dev/null (tree:783978eb212c1402ba2612bb32626e93af78b72d)
+++ e8194c62bfc68725972a6847fa2c6d529ca64137/objects.h (mode:100644 sha1:b6469e3e2ef191b36a23da23cbb8bf70ad54da0e)
@@ -0,0 +1,43 @@
+#ifndef OBJECTS_H
+#define OBJECTS_H
+
+/* Operations on the object database. */
+const char *sha1_file_directory;
+
+#define DB_ENVIRONMENT "SHA1_FILE_DIRECTORY"
+#define DEFAULT_DB_ENVIRONMENT ".git/objects"
+
+/* Takes a key and returns the contents, and sets *type to its type
+ * and size to its size.
+ */
+extern void * read_sha1_file(const unsigned char *sha1, char *type,
+ unsigned long *size);
+
+/* Writes the given data and length to a file, and returns the key. */
+extern int write_sha1_file(char *buf, unsigned len,
+ unsigned char *return_sha1);
+
+/* mmap the given file, and put its size in *size. */
+extern void * map_sha1_file(const unsigned char *sha1, unsigned long *size);
+
+/* Given a map, its size, and its expected type, return its contents and set
+ * *size to the size of the contents.
+ */
+extern void * unpack_sha1_file(void *map, unsigned long mapsize, char *type,
+ unsigned long *size);
+
+/* Return a statically allocated filename matching the sha1 signature */
+extern char *sha1_file_name(const unsigned char *sha1);
+
+/* Write a memory buffer out to the sha file */
+extern int write_sha1_buffer(const unsigned char *sha1, void *buf,
+ unsigned int size);
+/* compares the given buffer against its expected sha1 hash. */
+extern int check_sha1_signature(const unsigned char *sha1, void *buf,
+ unsigned long size);
+
+/* Convert to/from hex/sha1 representation */
+extern int get_sha1_hex(const char *hex, unsigned char *sha1);
+extern char * sha1_to_hex(const unsigned char *sha1); /* static buffer ret! */
+
+#endif /* OBJECTS_H */
Index: read-cache.c
===================================================================
--- 783978eb212c1402ba2612bb32626e93af78b72d/read-cache.c (mode:100644 sha1:7cdd0f82992789e64f6ea272e43ee4af8cdf7f2a)
+++ /dev/null (tree:e8194c62bfc68725972a6847fa2c6d529ca64137)
@@ -1,478 +0,0 @@
-/*
- * GIT - The information manager from hell
- *
- * Copyright (C) Linus Torvalds, 2005
- */
-#include <stdarg.h>
-#include "cache.h"
-
-const char *sha1_file_directory = NULL;
-struct cache_entry **active_cache = NULL;
-unsigned int active_nr = 0, active_alloc = 0;
-
-void usage(const char *err)
-{
- fprintf(stderr, "usage: %s\n", err);
- exit(1);
-}
-
-static void report(const char *prefix, const char *err, va_list params)
-{
- fputs(prefix, stderr);
- vfprintf(stderr, err, params);
- fputs("\n", stderr);
-}
-
-void die(const char *err, ...)
-{
- va_list params;
-
- va_start(params, err);
- report("fatal: ", err, params);
- va_end(params);
- exit(1);
-}
-
-int error(const char *err, ...)
-{
- va_list params;
-
- va_start(params, err);
- report("error: ", err, params);
- va_end(params);
- return -1;
-}
-
-
-static unsigned hexval(char c)
-{
- if (c >= '0' && c <= '9')
- return c - '0';
- if (c >= 'a' && c <= 'f')
- return c - 'a' + 10;
- if (c >= 'A' && c <= 'F')
- return c - 'A' + 10;
- return ~0;
-}
-
-int get_sha1_hex(const char *hex, unsigned char *sha1)
-{
- int i;
- for (i = 0; i < 20; i++) {
- unsigned int val = (hexval(hex[0]) << 4) | hexval(hex[1]);
- if (val & ~0xff)
- return -1;
- *sha1++ = val;
- hex += 2;
- }
- return 0;
-}
-
-char * sha1_to_hex(const unsigned char *sha1)
-{
- static char buffer[50];
- static const char hex[] = "0123456789abcdef";
- char *buf = buffer;
- int i;
-
- for (i = 0; i < 20; i++) {
- unsigned int val = *sha1++;
- *buf++ = hex[val >> 4];
- *buf++ = hex[val & 0xf];
- }
- return buffer;
-}
-
-/*
- * NOTE! This returns a statically allocated buffer, so you have to be
- * careful about using it. Do a "strdup()" if you need to save the
- * filename.
- */
-char *sha1_file_name(const unsigned char *sha1)
-{
- int i;
- static char *name, *base;
-
- if (!base) {
- char *sha1_file_directory = getenv(DB_ENVIRONMENT) ? : DEFAULT_DB_ENVIRONMENT;
- int len = strlen(sha1_file_directory);
- base = malloc(len + 60);
- memcpy(base, sha1_file_directory, len);
- memset(base+len, 0, 60);
- base[len] = '/';
- base[len+3] = '/';
- name = base + len + 1;
- }
- for (i = 0; i < 20; i++) {
- static char hex[] = "0123456789abcdef";
- unsigned int val = sha1[i];
- char *pos = name + i*2 + (i > 0);
- *pos++ = hex[val >> 4];
- *pos = hex[val & 0xf];
- }
- return base;
-}
-
-int check_sha1_signature(const unsigned char *sha1, void *map, unsigned long size)
-{
- unsigned char real_sha1[20];
- SHA_CTX c;
-
- SHA1_Init(&c);
- SHA1_Update(&c, map, size);
- SHA1_Final(real_sha1, &c);
- return memcmp(sha1, real_sha1, 20) ? -1 : 0;
-}
-
-void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
-{
- char *filename = sha1_file_name(sha1);
- int fd = open(filename, O_RDONLY);
- struct stat st;
- void *map;
-
- if (fd < 0) {
- perror(filename);
- return NULL;
- }
- if (fstat(fd, &st) < 0) {
- close(fd);
- return NULL;
- }
- map = mmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
- close(fd);
- if (-1 == (int)(long)map)
- return NULL;
- *size = st.st_size;
- return map;
-}
-
-void * unpack_sha1_file(void *map, unsigned long mapsize, char *type, unsigned long *size)
-{
- int ret, bytes;
- z_stream stream;
- char buffer[8192];
- char *buf;
-
- /* Get the data stream */
- memset(&stream, 0, sizeof(stream));
- stream.next_in = map;
- stream.avail_in = mapsize;
- stream.next_out = buffer;
- stream.avail_out = sizeof(buffer);
-
- inflateInit(&stream);
- ret = inflate(&stream, 0);
- if (sscanf(buffer, "%10s %lu", type, size) != 2)
- return NULL;
-
- bytes = strlen(buffer) + 1;
- buf = malloc(*size);
- if (!buf)
- return NULL;
-
- memcpy(buf, buffer + bytes, stream.total_out - bytes);
- bytes = stream.total_out - bytes;
- if (bytes < *size && ret == Z_OK) {
- stream.next_out = buf + bytes;
- stream.avail_out = *size - bytes;
- while (inflate(&stream, Z_FINISH) == Z_OK)
- /* nothing */;
- }
- inflateEnd(&stream);
- return buf;
-}
-
-void * read_sha1_file(const unsigned char *sha1, char *type, unsigned long *size)
-{
- unsigned long mapsize;
- void *map, *buf;
-
- map = map_sha1_file(sha1, &mapsize);
- if (map) {
- buf = unpack_sha1_file(map, mapsize, type, size);
- munmap(map, mapsize);
- return buf;
- }
- return NULL;
-}
-
-int write_sha1_file(char *buf, unsigned len, unsigned char *returnsha1)
-{
- int size;
- char *compressed;
- z_stream stream;
- unsigned char sha1[20];
- SHA_CTX c;
-
- /* Set it up */
- memset(&stream, 0, sizeof(stream));
- deflateInit(&stream, Z_BEST_COMPRESSION);
- size = deflateBound(&stream, len);
- compressed = malloc(size);
-
- /* Compress it */
- stream.next_in = buf;
- stream.avail_in = len;
- stream.next_out = compressed;
- stream.avail_out = size;
- while (deflate(&stream, Z_FINISH) == Z_OK)
- /* nothing */;
- deflateEnd(&stream);
- size = stream.total_out;
-
- /* Sha1.. */
- SHA1_Init(&c);
- SHA1_Update(&c, compressed, size);
- SHA1_Final(sha1, &c);
-
- if (write_sha1_buffer(sha1, compressed, size) < 0)
- return -1;
- if (returnsha1)
- memcpy(returnsha1, sha1, 20);
- return 0;
-}
-
-int write_sha1_buffer(const unsigned char *sha1, void *buf, unsigned int size)
-{
- char *filename = sha1_file_name(sha1);
- int fd;
-
- fd = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);
- if (fd < 0) {
- void *map;
-
- if (errno != EEXIST)
- return -1;
- /* #define NO_COLLISION_CHECK to possibly speed things up,
- * but the speedup should be really negligible. */
-#ifndef NO_COLLISION_CHECK
- fd = open(filename, O_RDONLY);
- if (fd < 0)
- return -1;
- map = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);
- if (map == MAP_FAILED)
- return -1;
- if (memcmp(buf, map, size))
- return error("SHA1 collision detected!"
- " This is bad, bad, BAD!\a\n");
-#endif
- return 0;
- }
- write(fd, buf, size);
- close(fd);
- return 0;
-}
-
-int cache_match_stat(struct cache_entry *ce, struct stat *st)
-{
- unsigned int changed = 0;
-
- /* nsec seems unreliable - not all filesystems support it, so
- * as long as it is in the inode cache you get right nsec
- * but after it gets flushed, you get zero nsec. */
- if (ce->mtime.sec != (unsigned int)st->st_mtim.tv_sec
-#ifdef NSEC
- || ce->mtime.nsec != (unsigned int)st->st_mtim.tv_nsec
-#endif
- )
- changed |= MTIME_CHANGED;
- if (ce->ctime.sec != (unsigned int)st->st_ctim.tv_sec
-#ifdef NSEC
- || ce->ctime.nsec != (unsigned int)st->st_ctim.tv_nsec
-#endif
- )
- changed |= CTIME_CHANGED;
- if (ce->st_uid != (unsigned int)st->st_uid ||
- ce->st_gid != (unsigned int)st->st_gid)
- changed |= OWNER_CHANGED;
- if (ce->st_mode != (unsigned int)st->st_mode)
- changed |= MODE_CHANGED;
- if (ce->st_dev != (unsigned int)st->st_dev ||
- ce->st_ino != (unsigned int)st->st_ino)
- changed |= INODE_CHANGED;
- if (ce->st_size != (unsigned int)st->st_size)
- changed |= DATA_CHANGED;
- return changed;
-}
-
-int cache_name_compare(const char *name1, int len1, const char *name2, int len2)
-{
- int len = len1 < len2 ? len1 : len2;
- int cmp;
-
- cmp = memcmp(name1, name2, len);
- if (cmp)
- return cmp;
- if (len1 < len2)
- return -1;
- if (len1 > len2)
- return 1;
- return 0;
-}
-
-int cache_name_pos(const char *name, int namelen)
-{
- int first, last;
-
- first = 0;
- last = active_nr;
- while (last > first) {
- int next = (last + first) >> 1;
- struct cache_entry *ce = active_cache[next];
- int cmp = cache_name_compare(name, namelen, ce->name, ce->namelen);
- if (!cmp)
- return next;
- if (cmp < 0) {
- last = next;
- continue;
- }
- first = next+1;
- }
- return -first-1;
-}
-
-int remove_file_from_cache(char *path)
-{
- int pos = cache_name_pos(path, strlen(path));
- if (pos >= 0) {
- active_nr--;
- if (pos < active_nr)
- memmove(active_cache + pos, active_cache + pos + 1, (active_nr - pos) * sizeof(struct cache_entry *));
- }
- return 0;
-}
-
-int add_cache_entry(struct cache_entry *ce, int ok_to_add)
-{
- int pos;
-
- pos = cache_name_pos(ce->name, ce->namelen);
-
- /* existing match? Just replace it */
- if (pos >= 0) {
- active_cache[pos] = ce;
- return 0;
- }
- pos = -pos-1;
-
- if (!ok_to_add)
- return -1;
-
- /* Make sure the array is big enough .. */
- if (active_nr == active_alloc) {
- active_alloc = alloc_nr(active_alloc);
- active_cache = realloc(active_cache, active_alloc * sizeof(struct cache_entry *));
- }
-
- /* Add it in.. */
- active_nr++;
- if (active_nr > pos)
- memmove(active_cache + pos + 1, active_cache + pos, (active_nr - pos - 1) * sizeof(ce));
- active_cache[pos] = ce;
- return 0;
-}
-
-static int verify_hdr(struct cache_header *hdr, unsigned long size)
-{
- SHA_CTX c;
- unsigned char sha1[20];
-
- if (hdr->signature != CACHE_SIGNATURE)
- return error("bad signature");
- if (hdr->version != 1)
- return error("bad version");
- SHA1_Init(&c);
- SHA1_Update(&c, hdr, offsetof(struct cache_header, sha1));
- SHA1_Update(&c, hdr+1, size - sizeof(*hdr));
- SHA1_Final(sha1, &c);
- if (memcmp(sha1, hdr->sha1, 20))
- return error("bad header sha1");
- return 0;
-}
-
-int read_cache(void)
-{
- int fd, i;
- struct stat st;
- unsigned long size, offset;
- void *map;
- struct cache_header *hdr;
-
- errno = EBUSY;
- if (active_cache)
- return error("more than one cachefile");
- errno = ENOENT;
- sha1_file_directory = getenv(DB_ENVIRONMENT);
- if (!sha1_file_directory)
- sha1_file_directory = DEFAULT_DB_ENVIRONMENT;
- if (access(sha1_file_directory, X_OK) < 0)
- return error("no access to SHA1 file directory");
- fd = open(".git/index", O_RDONLY);
- if (fd < 0)
- return (errno == ENOENT) ? 0 : error("open failed");
-
- size = 0; // avoid gcc warning
- map = (void *)-1;
- if (!fstat(fd, &st)) {
- size = st.st_size;
- errno = EINVAL;
- if (size >= sizeof(struct cache_header))
- map = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);
- }
- close(fd);
- if (-1 == (int)(long)map)
- return error("mmap failed");
-
- hdr = map;
- if (verify_hdr(hdr, size) < 0)
- goto unmap;
-
- active_nr = hdr->entries;
- active_alloc = alloc_nr(active_nr);
- active_cache = calloc(active_alloc, sizeof(struct cache_entry *));
-
- offset = sizeof(*hdr);
- for (i = 0; i < hdr->entries; i++) {
- struct cache_entry *ce = map + offset;
- offset = offset + ce_size(ce);
- active_cache[i] = ce;
- }
- return active_nr;
-
-unmap:
- munmap(map, size);
- errno = EINVAL;
- return error("verify header failed");
-}
-
-int write_cache(int newfd, struct cache_entry **cache, int entries)
-{
- SHA_CTX c;
- struct cache_header hdr;
- int i;
-
- hdr.signature = CACHE_SIGNATURE;
- hdr.version = 1;
- hdr.entries = entries;
-
- SHA1_Init(&c);
- SHA1_Update(&c, &hdr, offsetof(struct cache_header, sha1));
- for (i = 0; i < entries; i++) {
- struct cache_entry *ce = cache[i];
- int size = ce_size(ce);
- SHA1_Update(&c, ce, size);
- }
- SHA1_Final(hdr.sha1, &c);
-
- if (write(newfd, &hdr, sizeof(hdr)) != sizeof(hdr))
- return -1;
-
- for (i = 0; i < entries; i++) {
- struct cache_entry *ce = cache[i];
- int size = ce_size(ce);
- if (write(newfd, ce, size) != size)
- return -1;
- }
- return 0;
-}
Index: util.h
===================================================================
--- /dev/null (tree:783978eb212c1402ba2612bb32626e93af78b72d)
+++ e8194c62bfc68725972a6847fa2c6d529ca64137/util.h (mode:100644 sha1:d81ccd0b3363ff2a883e8d9e543af043728fd32a)
@@ -0,0 +1,9 @@
+#ifndef UTIL_H
+#define UTIL_H
+
+/* General helper functions */
+extern void usage(const char *err);
+extern void die(const char *err, ...);
+extern int error(const char *err, ...);
+
+#endif
^ permalink raw reply
* [PATCH] Clean up Makefile
From: Daniel Barkalow @ 2005-04-13 21:55 UTC (permalink / raw)
To: Petr Baudis; +Cc: git
In-Reply-To: <Pine.LNX.4.21.0504131556510.30848-100000@iabervon.org>
Use a pattern rule for building executables, and a list of common objects
(which could be made into a library).
Signed-Off-By: Daniel Barkalow <barkalow@iabervon.org>
Index: Makefile
===================================================================
--- 833bb44b8e49daf0424f44cf300748da17945859/Makefile (mode:100644 sha1:929aa49a3dbe683ad52094099797bc636a7949a6)
+++ 783978eb212c1402ba2612bb32626e93af78b72d/Makefile (mode:100644 sha1:2b626ec05b2e29016d26b5cb7b4f82ec5c4e79d6)
@@ -20,6 +20,8 @@
gitmerge.sh gitpull.sh gitrm.sh gittag.sh gittrack.sh gitexport.sh \
gitapply.sh
+COMMON= read-cache.o
+
GEN_SCRIPT= gitversion.sh
VERSION= VERSION
@@ -31,46 +33,8 @@
LIBS= -lssl -lz
-init-db: init-db.o
-
-update-cache: update-cache.o read-cache.o
- $(CC) $(CFLAGS) -o update-cache update-cache.o read-cache.o $(LIBS)
-
-show-diff: show-diff.o read-cache.o
- $(CC) $(CFLAGS) -o show-diff show-diff.o read-cache.o $(LIBS)
-
-write-tree: write-tree.o read-cache.o
- $(CC) $(CFLAGS) -o write-tree write-tree.o read-cache.o $(LIBS)
-
-read-tree: read-tree.o read-cache.o
- $(CC) $(CFLAGS) -o read-tree read-tree.o read-cache.o $(LIBS)
-
-commit-tree: commit-tree.o read-cache.o
- $(CC) $(CFLAGS) -o commit-tree commit-tree.o read-cache.o $(LIBS)
-
-cat-file: cat-file.o read-cache.o
- $(CC) $(CFLAGS) -o cat-file cat-file.o read-cache.o $(LIBS)
-
-ls-tree: ls-tree.o read-cache.o
- $(CC) $(CFLAGS) -o ls-tree ls-tree.o read-cache.o $(LIBS)
-
-checkout-cache: checkout-cache.o read-cache.o
- $(CC) $(CFLAGS) -o checkout-cache checkout-cache.o read-cache.o $(LIBS)
-
-diff-tree: diff-tree.o read-cache.o
- $(CC) $(CFLAGS) -o diff-tree diff-tree.o read-cache.o $(LIBS)
-
-fsck-cache: fsck-cache.o read-cache.o
- $(CC) $(CFLAGS) -o fsck-cache fsck-cache.o read-cache.o $(LIBS)
-
-rev-tree: rev-tree.o read-cache.o
- $(CC) $(CFLAGS) -o rev-tree rev-tree.o read-cache.o $(LIBS)
-
-show-files: show-files.o read-cache.o
- $(CC) $(CFLAGS) -o show-files show-files.o read-cache.o $(LIBS)
-
-check-files: check-files.o read-cache.o
- $(CC) $(CFLAGS) -o check-files check-files.o read-cache.o $(LIBS)
+$(PROG):%: %.o $(COMMON)
+ $(CC) $(CFLAGS) -o $@ $^ $(LIBS)
read-cache.o: cache.h
show-diff.o: cache.h
-- -Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Various cleanups
From: Daniel Barkalow @ 2005-04-13 21:52 UTC (permalink / raw)
To: Petr Baudis; +Cc: git
In-Reply-To: <Pine.LNX.4.21.0504131547330.30848-100000@iabervon.org>
As it turns out, I ended up with two patches: one to set up the Makefile,
and one to reorganize the common code. I'll do another one later to split
stuff out of rev-tree/fsck-cache once merge-base is in.
The reorganization depends on the Makefile.
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: Index/hash order
From: Florian Weimer @ 2005-04-13 21:40 UTC (permalink / raw)
To: Linus Torvalds; +Cc: H. Peter Anvin, Ingo Molnar, git
In-Reply-To: <Pine.LNX.4.58.0504131008500.4501@ppc970.osdl.org>
* Linus Torvalds:
> - I want things to distribute well. This means that it has to be based
> on a "append data" model, where historical data never changes, and you
> only append on top of it (either by adding totally new files, or by
> just letting the files grow).
Yes, I think this is something which can easily dominate the choice of
data structure.
> This works in a forward-delta environment (which is fundamentally based
> on the notion of "we know the old version, we're adding new stuff on
> top of it"), but does _not_ work in the backwards model of "we keep the
> old history as a delta against the new" model.
Forward deltas don't have to be terribly inefficient. You can get
O(log n) access to revision n fairly easily, using the trick described
there:
<http://svn.collab.net/repos/svn/trunk/notes/skip-deltas>
I've run a few tests, just to get a few numbers of the overhead
involved. I used the last ~8,000 changesets from the BKCVS kernel
repository. With cold cache, a checkout from cold cache takes about
250 seconds on my laptop. I don't have git numbers, but a mere copy
of the kernel tree needs 40 seconds.
For the hot-cache case, the difference is 140 seconds vs. 2.5 seconds
(or 6 seconds with decompression).
Uh-oh. I wouldn't have imaged the difference would be *that*
dramatic. The file system layer is *fast*.
Subversion's delta implementation is not a speed daemon (it handles
arbitrarily large files, which increases complexity significantly and
slows things down, compared to simpler in-memory algorithms), but it
will be very hard to come even close to the 2.5 seconds.
On the storage front, we have 220 MB for the skip deltas vs. 106 MB
for pure deltas-to-previous vs. 1.1 GB for uncompressed files
(directories are always delta-compressed, so to speak[1]). In the
first two cases, the first revision in the repository is deltaed
against /dev/null and itself and thus compressed, in case you think
the numbers are suspiciously low.
1. AFAICS, you can't really avoid that if you want to track file
identity information without introducing arbitrary file IDs.
^ permalink raw reply
* Re: Index/hash order
From: Linus Torvalds @ 2005-04-13 21:04 UTC (permalink / raw)
To: Ingo Molnar; +Cc: H. Peter Anvin, git
In-Reply-To: <20050413201523.GC27088@elte.hu>
On Wed, 13 Apr 2005, Ingo Molnar wrote:
>
> well, the 'owned by another user' solution is valid though, and doesnt
> have this particular problem. (We've got a secure multiuser OS, so can
> as well use it to protect the DB against corruption.)
So now you need root to set up new repositories? No thanks.
Linus
^ permalink raw reply
* Re: Index/hash order
From: Baruch Even @ 2005-04-13 20:28 UTC (permalink / raw)
To: Ingo Molnar; +Cc: Linus Torvalds, H. Peter Anvin, git
In-Reply-To: <20050413182909.GA25221@elte.hu>
Ingo Molnar wrote:
> with a plaintext repository we could do the 'hardlink trick' (which
> brings in other manageability problems and limitations but is at least a
> partially good idea), which would make the working tree and the
> repository share the same inode in most cases.
>
> While in the compressed case we'd have a separate compressed inode
> (taking up RAM with all its contents) and the working directory inode
> (taking up RAM) - summing up to more RAM than if we only had a single
> inode per object.
>
> furthermore, when generating/destroying large trees (which is a quite
> common thing), a hardlinked solution is faster, as it doesnt create
> 250MB+ of dirty RAM. In some cases (e.g. handling dozens of 'merge
> trees') it's dramatically faster.
You could still have the hardlink way by way of a .git/cache that keeps
uncompressed files, keep the files with their hash names but uncompressed.
It will be easy to find, fully hard-linkable, only keep the needed files
uncompressed and the three year old file compressed. The
You can even save some CPU time by checking if the file is in the cache
before decompressing it, though it does cost you with an extra disk
access to see if it's there or not. If you repeat the operation enough
you'll have the uncompressed version in the cache most of the times anyway.
Clear the cache weekly or so to avoid stale files from an ancient version.
Baruch
^ permalink raw reply
* Updated base64 patches
From: H. Peter Anvin @ 2005-04-13 20:26 UTC (permalink / raw)
To: Ingo Molnar; +Cc: Linus Torvalds, git
In-Reply-To: <20050413202102.GA27733@elte.hu>
I have uploaded two new base64 patches, one which uses the flat
repository and one which doesn't:
ftp://ftp.kernel.org/pub/linux/kernel/people/hpa/git-0.04-base64-3.diff
ftp://ftp.kernel.org/pub/linux/kernel/people/hpa/git-0.04-base64-flat-3.diff
... both are still against the git-0.04 tarball.
The only differences is changing "char" to "signed char" in places where
it actually matters (since plain char is unsigned on some platforms),
and, for the non-flat version, allowing the cache subdirectories to be
lazily created (if ENOENT is returned, try mkdir before giving up.)
-=hpa
^ permalink raw reply
* Re: Index/hash order
From: Ingo Molnar @ 2005-04-13 20:21 UTC (permalink / raw)
To: H. Peter Anvin; +Cc: Linus Torvalds, git
In-Reply-To: <20050413201836.GA27694@elte.hu>
* Ingo Molnar <mingo@elte.hu> wrote:
> > > I think abusing the immutable bit quickly will decend into the same
> > > rathole which makes u-w often useless. u-w will actually be preserved
> > > by more tools -- simply because they know about it -- than +i.
> >
> > well, the 'owned by another user' solution is valid though, and doesnt
> > have this particular problem. (We've got a secure multiuser OS, so can
> > as well use it to protect the DB against corruption.)
>
> but ... this variant doesnt have any 'wow' feeling to it either, and
> it clearly brings in a number of other limitations. I might as well
> shut up until i can suggest something obviously superior :)
i think the killer argument is compression. A 2 GB compressed repository
will be a hard sell already, 4 GB is pretty much out of question. And
once we accept that we have to have _some_ form of compression, it's
Linus' scheme that wins.
Ingo
^ permalink raw reply
* Re: Index/hash order
From: Ingo Molnar @ 2005-04-13 20:18 UTC (permalink / raw)
To: H. Peter Anvin; +Cc: Linus Torvalds, git
In-Reply-To: <20050413201523.GC27088@elte.hu>
* Ingo Molnar <mingo@elte.hu> wrote:
> > I think abusing the immutable bit quickly will decend into the same
> > rathole which makes u-w often useless. u-w will actually be preserved
> > by more tools -- simply because they know about it -- than +i.
>
> well, the 'owned by another user' solution is valid though, and doesnt
> have this particular problem. (We've got a secure multiuser OS, so can
> as well use it to protect the DB against corruption.)
but ... this variant doesnt have any 'wow' feeling to it either, and it
clearly brings in a number of other limitations. I might as well shut up
until i can suggest something obviously superior :)
Ingo
^ permalink raw reply
* Re: Index/hash order
From: Ingo Molnar @ 2005-04-13 20:15 UTC (permalink / raw)
To: H. Peter Anvin; +Cc: Linus Torvalds, git
In-Reply-To: <425D7C0F.2050109@zytor.com>
* H. Peter Anvin <hpa@zytor.com> wrote:
> >that's what i loosely meant under 'manageability problems'.
> >
> >I mentioned one solution earlier: to make the repository object an
> >immutable file (the +i flag on the inode) - it really wants to be
> >immutable after all. That would solve a whole range of 'accidental
> >corruption' issues.
> >
>
> I think abusing the immutable bit quickly will decend into the same
> rathole which makes u-w often useless. u-w will actually be preserved
> by more tools -- simply because they know about it -- than +i.
well, the 'owned by another user' solution is valid though, and doesnt
have this particular problem. (We've got a secure multiuser OS, so can
as well use it to protect the DB against corruption.)
> Either which way, it feels to me that this idea has already been ruled
> out, so it's probably pointless to keep debating just exactly what
> we're not actually going to do.
(even if it sounds stupid, i keep discussing decisions that are done for
reasons i cannot fully agree with (yet), even if i happen to agree with
the net decision. It's all technological arguments, so it's not like
there's anything fuzzy about any of these issues.)
Ingo
^ permalink raw reply
* Re: Index/hash order
From: Linus Torvalds @ 2005-04-13 20:15 UTC (permalink / raw)
To: H. Peter Anvin; +Cc: Ingo Molnar, git
In-Reply-To: <425D7C0F.2050109@zytor.com>
On Wed, 13 Apr 2005, H. Peter Anvin wrote:
>
> Either which way, it feels to me that this idea has already been ruled
> out, so it's probably pointless to keep debating just exactly what we're
> not actually going to do.
Hey, isn't that how most discussions progress? ;)
I don't mind alternatives per se. I'm just lazy. I came up with one
solution to the issues I percieved, and I like that one. But dammit, if
somebody comes up with something _clearly_ superior, I'll just bow down in
your general direction, and promptly implement that.
Linus
^ permalink raw reply
* Re: Index/hash order
From: H. Peter Anvin @ 2005-04-13 20:07 UTC (permalink / raw)
To: Ingo Molnar; +Cc: Linus Torvalds, git
In-Reply-To: <20050413200237.GA26635@elte.hu>
Ingo Molnar wrote:
>
> that's what i loosely meant under 'manageability problems'.
>
> I mentioned one solution earlier: to make the repository object an
> immutable file (the +i flag on the inode) - it really wants to be
> immutable after all. That would solve a whole range of 'accidental
> corruption' issues.
>
I think abusing the immutable bit quickly will decend into the same
rathole which makes u-w often useless. u-w will actually be preserved
by more tools -- simply because they know about it -- than +i.
Either which way, it feels to me that this idea has already been ruled
out, so it's probably pointless to keep debating just exactly what we're
not actually going to do.
-hpa
^ permalink raw reply
* Re: Index/hash order
From: Ingo Molnar @ 2005-04-13 20:02 UTC (permalink / raw)
To: Linus Torvalds; +Cc: H. Peter Anvin, git
In-Reply-To: <Pine.LNX.4.58.0504131144160.4501@ppc970.osdl.org>
* Linus Torvalds <torvalds@osdl.org> wrote:
> > with a plaintext repository we could do the 'hardlink trick' (which
> > brings in other manageability problems and limitations but is at least a
> > partially good idea), which would make the working tree and the
> > repository share the same inode in most cases.
> However, the real issue is that you're really asking for trouble.
> There are tons of tools that modify files without breaking the
> hardlink. Even some editors do. So you just use the wrong tool on the
> tree by mistake, and not only is your archive corrupt, you've
> corrupted all other archives that might have shared the same object
> directory.
that's what i loosely meant under 'manageability problems'.
I mentioned one solution earlier: to make the repository object an
immutable file (the +i flag on the inode) - it really wants to be
immutable after all. That would solve a whole range of 'accidental
corruption' issues.
Another solution (suggested by Christer Weinigel) was to enforce
immutability by making it owned by another user/group (git:git or
whatever).
but having a binary compressed format is 'soft immutability', done
cleverly.
Ingo
^ permalink raw reply
page: | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox