* Re: [PATCH] Skip writing out sha1 files for objects in packed git.
From: Linus Torvalds @ 2005-06-28 15:45 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vekanyx33.fsf@assigned-by-dhcp.cox.net>
On Mon, 27 Jun 2005, Junio C Hamano wrote:
>
> LT> And if we have a pack-file in .git/objects/ that already has
> LT> the object, that may not be the _same_ pack-file that we're
> LT> expanding at all, so if that pack file already has the
> LT> object, then not writing it out is actually the right thing
> LT> to do.
>
> This I have to think about a bit.
The most trivial example is doing a "git pull" of a small pack-file
update.
We probably don't want to leave it around as a pack-file (we'll re-pack
everything at some later date, but we also don't want to expand the stuff
we already have in our _real_ pack-file).
Linus
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Linus Torvalds @ 2005-06-28 15:49 UTC (permalink / raw)
To: Daniel Barkalow; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.21.0506280049090.30848-100000@iabervon.org>
On Tue, 28 Jun 2005, Daniel Barkalow wrote:
>
> It shouldn't be hard to add; the main issue is determining when
> transfering a pack file is a good idea, because it probably doesn't make
> sense to transfer a pack file just because the source side has an object
> that the target side wants in that pack.
Oh, you'd never just transfer the whole big pack-file at all: you'd just
create a new one. And creatign a new one is just a matter of finding the
common parent, and then doing
git-rev-list --objects common..HEAD | git-pack-file .git/tmp-pack
and then you send the result to the other side..
Linus
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Linus Torvalds @ 2005-06-28 16:21 UTC (permalink / raw)
To: Daniel Barkalow; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.58.0506280846100.19755@ppc970.osdl.org>
On Tue, 28 Jun 2005, Linus Torvalds wrote:
>
> Oh, you'd never just transfer the whole big pack-file at all: you'd just
> create a new one. And creatign a new one is just a matter of finding the
> common parent, and then doing
>
> git-rev-list --objects common..HEAD | git-pack-file .git/tmp-pack
>
> and then you send the result to the other side..
To clarify: this also works with objects that are already in another
pack-file (now that Junio fixed the "get size of a deltified packed
entry"), so you can have any number of unpacked objects in your objects
directory, _and_ a pack-file (or several), and you can generate a new
temporary pack-file just for sending somewhere else that contains
arbistrary parts of that (ie a mix of objects that are in your "main"
packfiles and objects that are unpacked).
You don't have to use "git-rev-list" to generate the objects, btw,
git-pack-file takes an arbitrary list of object ID's (plus a "packing
hint" in the form of a filename that is not required, but that can help
the packing heuristics, and that git-rev-list does provide).
I'll also fix up git-pack-file to be able to pack tag objects (and the
unpacking to understand them), so that any valid object can be packed.
Right now it only handles the objects that git-rev-list knows about.
Linus
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Benjamin LaHaise @ 2005-06-28 16:35 UTC (permalink / raw)
To: Petr Baudis; +Cc: Christopher Li, Junio C Hamano, Linus Torvalds, git
In-Reply-To: <20050628145256.GA1275@pasky.ji.cz>
On Tue, Jun 28, 2005 at 04:52:56PM +0200, Petr Baudis wrote:
> I think the git-*-pull tools are actually just fine. You will only need
> to have some server-side CGI gadget to frontend the file, but we need
> that anyway to make the pull reasonably effective.
Not really -- the use of rsync for the objects fails horribly on slow
links when the project scales in the number of commits. The rsync
protocol has to transfer the names of each file and some information
about it, and that information isn't delta compressed. This is where
kernel.org is falling over, as well as what makes the kernel tree very
painful to use over a dialup modem link.
-ben
--
"Time is what keeps everything from happening all at once." -- John Wheeler
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Linus Torvalds @ 2005-06-28 16:45 UTC (permalink / raw)
To: Christopher Li; +Cc: Git Mailing List
In-Reply-To: <20050628103852.GB21533@64m.dyndns.org>
On Tue, 28 Jun 2005, Christopher Li wrote:
>
> That is all nice improvement to address the space usage issue.
>
> Should people just run repacking once a while or is it automaticly
> add new object to the pack file?
While adding a new object to a pack file is _possible_ (you add it to the
end of the pack-file, and re-generate the index file), I would strongly
suggest against it for several reasons:
- It's a lot more complex and expensive than just writing a new file.
Much better to make the pack generation be an off-line thing, and make
new object creation really cheap.
- it has serious locking issues, and if something goes wrong you are just
horribly screwed. This implies, for example, that to be safe you really
have to use fsync() etc at every point (and be careful about writing
the index), making the update even _more_ expensive. Over NFS you need
to be extremely careful to make sure that everybody got the right lock,
yadda yadda.
Packing things off-line just means that _all_ of these problems go
away.
- There are operations that want to remove objects (I do that all the
time: I do something stupid, and decide to undo it, or I just do a
"git-update-cache" and notice that I need to do more work so I edit it
some more and actually never commit the first version)
If _adding_ to the file had some serious correctness issues, _removing_
an object from a file is even worse. MUCH worse. Now you don't just
have to lock against other people creating new objects, now you have to
lock against updates (or totally re-write the whole big file and do an
atomic "rename").
- it can actually generate worse packing. The current "offline" method
means that we can pack any version of a file against any other version
of a file, and we do. We pick the closest version we can find, and we
try to always pack against the bigger one (deletes are smaller deltas,
and the biggest one tends to be the latest version, so this not only
means that the delta is denser, it also means that the latest version -
which is likely to be the biggest and most often used - tends to be
non-delta).
In contrast, updating the pack file means that you always write the
latest version as a delta, which means that you're doing things
_exactly_ the wrong way around both for performance and size.
- Finally: packing allows us to do optimize for locality. In particular,
I write out the pack file in "recency" order, ie the top-most objects
go first, and in particular, the "commit" objects go at the very top of
the file. Why? Because it means that the commit objects (which are
heavily used for the history generation by pretty much anything, since
"git-rev-list" will access them) are packed together, and in the right
order.
Again, you can't do that if you do on-line updates as opposed to
offline packing.
So the usage pattern I envision is to pack stuff maybe once a month
(depending on how much changes, of course), because then you really do get
the best of both worlds: the simplicity of individual objects for recent
work and the optimal packing and ordering that you can really work on for
the longer range case. And your project never grows very big.
Btw, I'm not claiming that my current pack format is "optimal" of course.
For example, while I write all objects in recency order, right now that
means that if a recent object has been written as a delta that depends on
an older one, I actually write the delta first (correct) but I won't write
the older object until its recency ordering (wrong).
That kind of thing is trivial to fix (eventually), but it's an example of
where ordering matters (ie if it's the other way around: the delta is the
older object, it's probably better to leave it at the end of the file,
since it's probably not going to be accessed much, making the effective
packing at the head more efficicient). It's also an example of the kinds
of things we can do exactly because we're doing the packing off-line.
Linus
^ permalink raw reply
* Cygwin and Native MS Windows (was: Mercurial vs Updated git HOWTO for kernel hackers)
From: Kevin Smith @ 2005-06-28 16:50 UTC (permalink / raw)
To: Petr Baudis; +Cc: mercurial, Git Mailing List
In-Reply-To: <20050628150752.GC1275@pasky.ji.cz>
Petr Baudis wrote:
> Dear diary, on Fri, Jun 24, 2005 at 03:57:00PM CEST, I got a letter
> where Kevin Smith <yarcs@qualitycode.com> told me that...
>
>>- Can run on (native) MS Windows
>> (necessary for me because I often work on cross-platform projects)
>
>
> I'd expect everything to work fine with Cygwin (or with only minor
> problems easy to fix) or just any working bash + GNU coreutils
> installation. Any issue with that?
I don't see cygwin as being a reasonable option for a core tool like
SCM. Cygwin is too large and invasive to expect Windows users to install
and use it. For my own projects, native Windows operation is a requirement.
Kevin
^ permalink raw reply
* Cogito vs. Git (was: Mercurial vs Updated git HOWTO for kernel hackers)
From: Kevin Smith @ 2005-06-28 16:51 UTC (permalink / raw)
To: Petr Baudis; +Cc: Git Mailing List
In-Reply-To: <20050628150752.GC1275@pasky.ji.cz>
Petr Baudis wrote:
> Cogito's only unusual requirement (well, expectation) is that HEAD is a
> symlink to .git/refs/heads/master, and .git/refs/heads/master should
> reflect your current head. I will try to ease up this restriction so
> that things will mostly work even if you just have HEAD. I think that
> most auxiliary commands (e.g. cg-log - you just have to love it) should
> work on any sensible git tree (but I didn't test it - yet).
So you're saying that aside from that one solveable issue, I could use
low-level git tools, third-party over-git tools, and cogito,
interchangebly on a single repo without the tools becoming confused?
That's cool, and should be made more clear in the cogito readme. I was
under the impression that cogito was tracking all kinds of extra meta
magic stuff that git tools wouldn't keep updated.
If I were using cogito, I probably wouldn't want to use the low-level
git stuff directly, but I might want to use (or maybe even write) some
other over-git tools.
You might also consider removing the "Core GIT" section from the README,
because I think it increases the confusion between the two.
Cheers,
Kevin
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Daniel Barkalow @ 2005-06-28 17:04 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.58.0506280903510.19755@ppc970.osdl.org>
On Tue, 28 Jun 2005, Linus Torvalds wrote:
> I'll also fix up git-pack-file to be able to pack tag objects (and the
> unpacking to understand them), so that any valid object can be packed.
> Right now it only handles the objects that git-rev-list knows about.
Actually, the ideal thing would be to move the packing code into an object
file that git-ssh-push can include; that way it can write directly to the
socket instead of going through disk, and it can also go from getting the
remote end's list of common ancestors to having a pack to send without
needing to exec a script.
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Linus Torvalds @ 2005-06-28 17:36 UTC (permalink / raw)
To: Daniel Barkalow; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.21.0506281251380.30848-100000@iabervon.org>
On Tue, 28 Jun 2005, Daniel Barkalow wrote:
>
> Actually, the ideal thing would be to move the packing code into an object
> file that git-ssh-push can include; that way it can write directly to the
> socket instead of going through disk
It doesn't work very easily that way because the index file (which
contains the object list and the offsets into the pack file) cannot be
created until after the pack file has been created (and we don't want to
evaluate that one in memory, since it can be quite big).
Now, what we could do is to stream out the pack file first to stdout, and
write the index file afterwards. But since we don't know how big the pack
file will be when we start packing, and the pack-file can contain
basically arbitrary patterns, that requires that the receiver actually
parse the pack-file as it comes in.
The format of the pack-file is a fairly trivial data stream of
- rinse and repeat for each object:
- one character of type of file (C, T, B, G, D for "commit", "tree",
"blob", "tag" or "delta" respectively)
- four bytes of network-order unpacked data length
- [ if delta: 20 bytes of delta object ID ]
- zlib-packed data (length unknown, except we know how much we want
it to unpack to)
- Finally at the end: 20 bytes of SHA1 of the pack-file contents (up to
the SHA1)
so it's actually possible to pick up the objects as they come off the
stream, since the SHA1 name is defined by the contents and you don't need
the index file unless you want to look things up.
So the receiver side could try this algorithm:
- unpack each object in memory on the receiving side
If the unpack failed, it must have been the SHA1 at the end, so
verify it!
- if it's a delta object and you haven't seen the object it's a delta
against, keep it in memory.
- if it's a non-delta object, just write it to the object store, and try
to resolve any delta objects you have pending that this new object
satisfies. That in turn creates other objects that may have more deltas
they satisfy etc.
which looks quite doable. The delta objects are small, so keeping them in
memory shouldn't be a problem (especially since we _tend_ to write deltas
after the object they depend on).
I can certainly add an option to git-pack-file that disables writing of
the index file, and just writes the pack-file to stdout. I'm not sure I
want to write the "parse incoming pack-file" thing, but git-unpack-objects
comes _reasonably_ close (but right now it seeks around using the index
file to resolve deltas, instead of keeping them in memory and resolving
them when possible). But I can make the infrastructure ready for it.
Sounds like a plan.
Linus
^ permalink raw reply
* Re: Mercurial vs Updated git HOWTO for kernel hackers
From: Matt Mackall @ 2005-06-28 18:01 UTC (permalink / raw)
To: Petr Baudis; +Cc: Jeff Garzik, Linux Kernel, Git Mailing List, mercurial
In-Reply-To: <20050628150027.GB1275@pasky.ji.cz>
On Tue, Jun 28, 2005 at 05:00:27PM +0200, Petr Baudis wrote:
> > Mercurial's undo is taking a snapshot of all the changed file's repo file length
> > at every commit or pull. It just truncate the file to original size and undo
> > is done.
>
> "Trunactes"? That sounds very wrong... you mean replace with old
> version? Anyway, what if the file has same length? It just doesn't make
> much sense to me.
Everything in Mercurial is an append-only log. A transaction journal
records the original length of each log so that it can be restored on
failure.
--
Mathematics is the supreme nostalgia of our time.
^ permalink raw reply
* Re: kernel.org and GIT tree rebuilding
From: Nicolas Pitre @ 2005-06-28 18:06 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Git Mailing List, Chris Mason
In-Reply-To: <Pine.LNX.4.58.0506260905200.19755@ppc970.osdl.org>
On Sun, 26 Jun 2005, Linus Torvalds wrote:
> On Fri, 24 Jun 2005, Linus Torvalds wrote:
> >
> > yeah, it clearly needs some refining to be useful, but I think you can
> > kind of see how it would work.
>
> Ok, here's how it works.
>
[...]
>
> Nico? Chris? Interested in giving it a look? It's kind of a combination of
> your things, generalized and then made to have fast lookup with the index.
Wow !
Look away for a few days and when you look back your own stuff has been
yanked out!
But actually I like this packing thing. Certainly much nicer to work
with and also more useful. And, since the pack file has its own
checksum it is possible to use that pack format (without the index
(which can be recreated from the pack file alone anyway) for network
transfer.
Here's one improvement to the pack format, breaking it early so it won't
affect anyone at this point: compressed object header. Instead of the
fixed 5 byte header, this patch convert it to a variable size granted
most object are small enough to save on the storage of the significant
size bytes which will be zero and packing the non-zero byte position
with the object type.
This can save up to 2% or 21KB on the test I
performed with the git repository.
diff --git a/pack-objects.c b/pack-objects.c
--- a/pack-objects.c
+++ b/pack-objects.c
@@ -7,7 +7,7 @@
static const char pack_usage[] = "git-pack-objects [--window=N] [--depth=N] base-name < object-list";
enum object_type {
- OBJ_NONE,
+ OBJ_NONE = 0,
OBJ_COMMIT,
OBJ_TREE,
OBJ_BLOB,
@@ -55,7 +55,7 @@ static unsigned long write_object(struct
char type[10];
void *buf = read_sha1_file(entry->sha1, type, &size);
char header[25];
- unsigned hdrlen, datalen;
+ unsigned hdrlen, datalen, bit;
if (!buf)
die("unable to read %s", sha1_to_hex(entry->sha1));
@@ -63,21 +63,33 @@ static unsigned long write_object(struct
die("object %s size inconsistency (%lu vs %lu)", sha1_to_hex(entry->sha1), size, entry->size);
/*
- * The object header is a byte of 'type' followed by four bytes of
- * length, except for deltas that has the 20 bytes of delta sha
- * instead.
+ * The object header is one byte which low 4 bits represent the
+ * object type, the 4 upper bits indicates which of the following
+ * bytes are used to build the object size. For delta objects the
+ * sha1 of the reference object is also appended.
*/
- header[0] = ".CTB"[entry->type];
- hdrlen = 5;
+ header[0] = entry->type;
if (entry->delta) {
- header[0] = 'D';
- memcpy(header+5, entry->delta, 20);
+ header[0] = OBJ_DELTA;
buf = delta_against(buf, size, entry);
size = entry->delta_size;
- hdrlen = 25;
}
- datalen = htonl(size);
- memcpy(header+1, &datalen, 4);
+ hdrlen = 1;
+ bit = (1 << 4);
+ datalen = size;
+ do {
+ if (datalen & 0xff) {
+ header[0] |= bit;
+ header[hdrlen++] = datalen;
+ }
+ bit <<= 1;
+ datalen >>= 8;
+ } while (datalen);
+ if (entry->delta) {
+ memcpy(header+hdrlen, entry->delta, 20);
+ hdrlen += 20;
+ }
+
sha1write(f, header, hdrlen);
datalen = sha1write_compressed(f, buf, size);
free(buf);
diff --git a/unpack-objects.c b/unpack-objects.c
--- a/unpack-objects.c
+++ b/unpack-objects.c
@@ -12,6 +12,16 @@ struct pack_entry {
unsigned char sha1[20];
};
+enum object_type {
+ OBJ_NONE = 0,
+ OBJ_COMMIT,
+ OBJ_TREE,
+ OBJ_BLOB,
+ OBJ_DELTA
+};
+
+static char *type_s[] = { NULL, "commit", "tree", "blob", "delta" };
+
static void *pack_base;
static unsigned long pack_size;
static void *index_base;
@@ -92,7 +102,7 @@ static int check_index(void)
}
static int unpack_non_delta_entry(struct pack_entry *entry,
- int kind,
+ char *type,
unsigned char *data,
unsigned long size,
unsigned long left)
@@ -101,9 +111,8 @@ static int unpack_non_delta_entry(struct
z_stream stream;
char *buffer;
unsigned char sha1[20];
- char *type_s;
- printf("%s %c %lu\n", sha1_to_hex(entry->sha1), kind, size);
+ printf("%s %s %lu\n", sha1_to_hex(entry->sha1), type, size);
if (dry_run)
return 0;
@@ -120,18 +129,12 @@ static int unpack_non_delta_entry(struct
inflateEnd(&stream);
if ((st != Z_STREAM_END) || stream.total_out != size)
goto err_finish;
- switch (kind) {
- case 'C': type_s = "commit"; break;
- case 'T': type_s = "tree"; break;
- case 'B': type_s = "blob"; break;
- default: goto err_finish;
- }
- if (write_sha1_file(buffer, size, type_s, sha1) < 0)
+ if (write_sha1_file(buffer, size, type, sha1) < 0)
die("failed to write %s (%s)",
- sha1_to_hex(entry->sha1), type_s);
- printf("%s %s\n", sha1_to_hex(sha1), type_s);
+ sha1_to_hex(entry->sha1), type);
+ printf("%s %s\n", sha1_to_hex(sha1), type);
if (memcmp(sha1, entry->sha1, 20))
- die("resulting %s have wrong SHA1", type_s);
+ die("resulting %s have wrong SHA1", type);
finish:
st = 0;
@@ -183,15 +186,13 @@ static int unpack_delta_entry(struct pac
die("truncated pack file");
data = base_sha1 + 20;
data_size = left - 20;
- printf("%s D %lu", sha1_to_hex(entry->sha1), delta_size);
+ printf("%s delta %lu", sha1_to_hex(entry->sha1), delta_size);
printf(" %s\n", sha1_to_hex(base_sha1));
if (dry_run)
return 0;
- /* pack+5 is the base sha1, unless we have it, we need to
- * unpack it first.
- */
+ /* unless we have the base sha1, we need to unpack it first. */
if (!has_sha1_file(base_sha1)) {
struct pack_entry *base;
if (!find_pack_entry(base_sha1, &base))
@@ -236,7 +237,9 @@ static int unpack_delta_entry(struct pac
static void unpack_entry(struct pack_entry *entry)
{
unsigned long offset, size, left;
- unsigned char *pack;
+ unsigned char *pack, sizebits;
+ enum object_type type;
+ int i;
/* Have we done this one already due to deltas based on it? */
if (lookup_object(entry->sha1))
@@ -246,14 +249,25 @@ static void unpack_entry(struct pack_ent
if (offset > pack_size - 5)
die("object offset outside of pack file");
pack = pack_base + offset;
- size = (pack[1] << 24) + (pack[2] << 16) + (pack[3] << 8) + pack[4];
- left = pack_size - offset - 5;
- switch (*pack) {
- case 'C': case 'T': case 'B':
- unpack_non_delta_entry(entry, *pack, pack+5, size, left);
+ sizebits = *pack++;
+ type = sizebits & 0x0f;
+ sizebits >>= 4;
+ i = size = 0;
+ while (sizebits) {
+ if (sizebits & 1)
+ size |= *pack++ << i;
+ i += 8;
+ sizebits >>= 1;
+ }
+ left = pack_size - ((void *)pack - pack_base);
+ switch (type) {
+ case OBJ_COMMIT:
+ case OBJ_TREE:
+ case OBJ_BLOB:
+ unpack_non_delta_entry(entry, type_s[type], pack, size, left);
break;
- case 'D':
- unpack_delta_entry(entry, pack+5, size, left);
+ case OBJ_DELTA:
+ unpack_delta_entry(entry, pack, size, left);
break;
default:
die("corrupted pack file");
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Linus Torvalds @ 2005-06-28 18:17 UTC (permalink / raw)
To: Daniel Barkalow; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.58.0506281019450.19755@ppc970.osdl.org>
On Tue, 28 Jun 2005, Linus Torvalds wrote:
>
> I can certainly add an option to git-pack-file that disables writing of
> the index file, and just writes the pack-file to stdout.
Done.
> I'm not sure I
> want to write the "parse incoming pack-file" thing, but git-unpack-objects
> comes _reasonably_ close (but right now it seeks around using the index
> file to resolve deltas, instead of keeping them in memory and resolving
> them when possible).
I'm still thinking about this one. I think I'll just do it.
One problem here is that since we don't know how big the incoming
pack-file will be, in a streaming input environment the receiver needs to
either make the pack-file reception be the last thing it sees, or it will
have to live with the fact that "git-unpack-objects" will read some more
than it needs before it notices that it got it all...
We can handle the latter either by padding (make the rule be that
git-unpack-file will always read in chunks of 4kB max, and pad the output
with 4kB of zero bytes or something, and then you can execute
git-unpack-objects and continue reading stdin afterwards, removing any
zeroes that git-unpack-file didn't eat), or by having git-unpack-objects
flush anything after the final SHA1 to _its_ stdout, so that you can get
the following data/commands in the stream from the unpack-file thing.
Ugly, in any case.
Linus
^ permalink raw reply
* GIT-CVS sync script
From: Panagiotis Issaris @ 2005-06-28 19:14 UTC (permalink / raw)
To: git; +Cc: torvalds
Hi,
I find the included script useful to easily track a CVS repository in
GIT. You can both call it for the initial import and ofcourse for the
subsequent resynchronisation with the parent CVS repository (that's what
the script's about).
Would it be better to provide this as a patch against git-cvsimport-script?
It has been tested on the RTAI-Fusion repository.
With friendly regards,
Takis
Signed-off-by: Panagiotis Issaris <takis@gna.org>
#!/usr/bin/env bash
#
# Keep a GIT repository in sync with a CVS repository.
#
# Copyright (c) Panagiotis Issaris, 2005.
#
CVS2GIT_PATCHSETNR=.cvs2git_patchsetnr
CVS2GIT_UPGRADESCRIPT=.cvs2git_updatescript
PROJECT_GITDIR=`pwd`
PROJECT_CVSDIR=$1
CVSPS_OPTIONS="--no-rlog"
usage () {
echo -e "Usage: \n\t$0 /path/to/project/cvsdir"
exit 1
}
if test $# -lt 1 ; then
usage
else
if test ! -d $PROJECT_CVSDIR; then
usage
fi
fi
cd $PROJECT_CVSDIR
if test ! -d $PROJECT_GITDIR/.git; then
TZ=UTC cvsps $CVSPS_OPTIONS -A > $PROJECT_GITDIR/.git-cvsps-result
git-cvs2git --cvsroot=`cat CVS/Root` --module=`cat CVS/Repository` < $PROJECT_GITDIR/.git-cvsps-result > $PROJECT_GITDIR/.cvs2git_updatescript
else
if test -f $PROJECT_GITDIR/.cvs2git_patchsetnr; then
PATCHSETNR=`cat $PROJECT_GITDIR/.cvs2git_patchsetnr`;
else
PATCHSETNR=`cvsps | grep "^PatchSet " | tail -n1 | cut -f2 -d' '`
fi
let PATCHSETNR=$PATCHSETNR+1
# Updating from patchset $PATCHSETNR...
TZ=UTC cvsps $CVSPS_OPTIONS -x -s $PATCHSETNR- -A > $PROJECT_GITDIR/.git-cvsps-result
if test ! -s $PROJECT_GITDIR/.git-cvsps-result; then
echo "No new patchsets available"
exit 2
fi
git-cvs2git -u --cvsroot=`cat CVS/Root` --module=`cat CVS/Repository` < $PROJECT_GITDIR/.git-cvsps-result > $PROJECT_GITDIR/.cvs2git_updatescript
fi
# Storing last patchset number $PATCHSETNR
PATCHSETNR=`grep "^PatchSet " $PROJECT_GITDIR/.git-cvsps-result | tail -n1 | cut -f2 -d' '`
echo $PATCHSETNR > $PROJECT_GITDIR/.cvs2git_patchsetnr
# And finally, execute the generated script
cd $PROJECT_GITDIR
sh .cvs2git_updatescript
^ permalink raw reply
* [PATCH] cvsimport: rewritten in Perl
From: Matthias Urlichs @ 2005-06-28 19:23 UTC (permalink / raw)
To: git
I just got my machine blocked from a CVS server which didn't like
to get hammered with connections.
That was cvs2git's shell script. Which, by the way, is slow as hell.
Appended: a git-cvsimport script, written in Perl, which directly talks
to the CVS server. If the repository is local, it runs a "cvs server"
child. It produces the same git repository as Linus' version. It can do
incremental imports. And it's 20 times faster (on my system, with a
local CVS repository).
cvs2git is thus obsolete; this patch deletes it.
Signed-Off-By: Matthias Urlichs <smurf@smurf.noris.de>
---
diff --git a/Documentation/cvs-migration.txt b/Documentation/cvs-migration.txt
--- a/Documentation/cvs-migration.txt
+++ b/Documentation/cvs-migration.txt
@@ -63,18 +63,38 @@ Once you've gotten (and installed) cvsps
any more familiar with it, but make sure it is in your path. After that,
the magic command line is
- git cvsimport <cvsroot> <module>
+ git cvsimport -v -d <cvsroot> <module> <destination>
which will do exactly what you'd think it does: it will create a git
-archive of the named CVS module. The new archive will be created in a
-subdirectory named <module>.
+archive of the named CVS module. The new archive will be created in the
+subdirectory named <destination>; it'll be created if it doesn't exist.
+Default is the local directory.
It can take some time to actually do the conversion for a large archive
since it involves checking out from CVS every revision of every file,
-and the conversion script can be reasonably chatty, but on some not very
-scientific tests it averaged about eight revisions per second, so a
-medium-sized project should not take more than a couple of minutes. For
-larger projects or remote repositories, the process may take longer.
+and the conversion script is reasonably chatty unless you omit the '-v'
+option, but on some not very scientific tests it averaged about twenty
+revisions per second, so a medium-sized project should not take more
+than a couple of minutes. For larger projects or remote repositories,
+the process may take longer.
+
+After the (initial) import is done, the CVS archive's current head
+revision will be checked out -- thus, you can start adding your own
+changes right away.
+
+The import is incremental, i.e. if you call it again next month it'll
+fetch any CVS updates that have been happening in the meantime. The
+cut-off is date-based, so don't change the branches that were imported
+from CVS.
+
+You can merge those updates (or, in fact, a different CVS branch) into
+your main branch:
+
+ cg-merge <branch>
+
+The HEAD revision from CVS is named "origin", not "HEAD", because git
+already uses "HEAD". (If you don't like 'origin', use cvsimport's
+'-o' option to change it.)
Emulating CVS behaviour
diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -35,7 +35,7 @@ PROG= git-update-cache git-diff-files
git-http-pull git-ssh-push git-ssh-pull git-rev-list git-mktag \
git-diff-helper git-tar-tree git-local-pull git-write-blob \
git-get-tar-commit-id git-apply git-stripspace \
- git-cvs2git git-diff-stages git-rev-parse git-patch-id \
+ git-diff-stages git-rev-parse git-patch-id \
git-pack-objects git-unpack-objects
all: $(PROG)
@@ -118,7 +118,6 @@ git-diff-helper: diff-helper.c
git-tar-tree: tar-tree.c
git-write-blob: write-blob.c
git-stripspace: stripspace.c
-git-cvs2git: cvs2git.c
git-diff-stages: diff-stages.c
git-rev-parse: rev-parse.c
git-patch-id: patch-id.c
diff --git a/cvs2git.c b/cvs2git.c
deleted file mode 100644
--- a/cvs2git.c
+++ /dev/null
@@ -1,329 +0,0 @@
-/*
- * cvs2git
- *
- * Copyright (C) Linus Torvalds 2005
- */
-
-#include <stdio.h>
-#include <ctype.h>
-#include <string.h>
-#include <stdlib.h>
-#include <unistd.h>
-
-static int verbose = 0;
-
-/*
- * This is a really stupid program that takes cvsps output, and
- * generates a a long _shell_script_ that will create the GIT archive
- * from it.
- *
- * You've been warned. I told you it was stupid.
- *
- * NOTE NOTE NOTE! In order to do branches correctly, this needs
- * the fixed cvsps that has the "Ancestor branch" tag output.
- * Hopefully David Mansfield will update his distribution soon
- * enough (he's the one who wrote the patch, so at least we don't
- * have to figt maintainer issues ;)
- *
- * Usage:
- *
- * TZ=UTC cvsps -A |
- * git-cvs2git --cvsroot=[root] --module=[module] > script
- *
- * Creates a shell script that will generate the .git archive of
- * the names CVS repository.
- *
- * TZ=UTC cvsps -s 1234- -A |
- * git-cvs2git -u --cvsroot=[root] --module=[module] > script
- *
- * Creates a shell script that will update the .git archive with
- * CVS changes from patchset 1234 until the last one.
- *
- * IMPORTANT NOTE ABOUT "cvsps"! This requires version 2.1 or better,
- * and the "TZ=UTC" and the "-A" flag is required for sane results!
- */
-enum state {
- Header,
- Log,
- Members
-};
-
-static const char *cvsroot;
-static const char *cvsmodule;
-
-static char date[100];
-static char author[100];
-static char branch[100];
-static char ancestor[100];
-static char tag[100];
-static char log[32768];
-static int loglen = 0;
-static int initial_commit = 1;
-
-static void lookup_author(char *n, char **name, char **email)
-{
- /*
- * FIXME!!! I'm lazy and stupid.
- *
- * This could be something like
- *
- * printf("lookup_author '%s'\n", n);
- * *name = "$author_name";
- * *email = "$author_email";
- *
- * and that would allow the script to do its own
- * lookups at run-time.
- */
- *name = n;
- *email = n;
-}
-
-static void prepare_commit(void)
-{
- char *author_name, *author_email;
- char *src_branch;
-
- lookup_author(author, &author_name, &author_email);
-
- printf("export GIT_COMMITTER_NAME=%s\n", author_name);
- printf("export GIT_COMMITTER_EMAIL=%s\n", author_email);
- printf("export GIT_COMMITTER_DATE='+0000 %s'\n", date);
-
- printf("export GIT_AUTHOR_NAME=%s\n", author_name);
- printf("export GIT_AUTHOR_EMAIL=%s\n", author_email);
- printf("export GIT_AUTHOR_DATE='+0000 %s'\n", date);
-
- if (initial_commit)
- return;
-
- src_branch = *ancestor ? ancestor : branch;
- if (!strcmp(src_branch, "HEAD"))
- src_branch = "master";
- printf("ln -sf refs/heads/'%s' .git/HEAD\n", src_branch);
-
- /*
- * Even if cvsps claims an ancestor, we'll let the new
- * branch name take precedence if it already exists
- */
- if (*ancestor) {
- src_branch = branch;
- if (!strcmp(src_branch, "HEAD"))
- src_branch = "master";
- printf("[ -e .git/refs/heads/'%s' ] && ln -sf refs/heads/'%s' .git/HEAD\n",
- src_branch, src_branch);
- }
-
- printf("git-read-tree -m HEAD || exit 1\n");
- printf("git-checkout-cache -f -u -a\n");
-}
-
-static void commit(void)
-{
- const char *cmit_parent = initial_commit ? "" : "-p HEAD";
- const char *dst_branch;
- char *space;
- int i;
-
- printf("tree=$(git-write-tree)\n");
- printf("cat > .cmitmsg <<EOFMSG\n");
-
- /* Escape $ characters, and remove control characters */
- for (i = 0; i < loglen; i++) {
- unsigned char c = log[i];
-
- switch (c) {
- case '$':
- case '\\':
- case '`':
- putchar('\\');
- break;
- case 0 ... 31:
- if (c == '\n' || c == '\t')
- break;
- case 128 ... 159:
- continue;
- }
- putchar(c);
- }
- printf("\nEOFMSG\n");
- printf("commit=$(cat .cmitmsg | git-commit-tree $tree %s)\n", cmit_parent);
-
- dst_branch = branch;
- if (!strcmp(dst_branch, "HEAD"))
- dst_branch = "master";
-
- printf("echo $commit > .git/refs/heads/'%s'\n", dst_branch);
-
- space = strchr(tag, ' ');
- if (space)
- *space = 0;
- if (strcmp(tag, "(none)"))
- printf("echo $commit > .git/refs/tags/'%s'\n", tag);
-
- printf("echo 'Committed (to %s):' ; cat .cmitmsg; echo\n", dst_branch);
-
- *date = 0;
- *author = 0;
- *branch = 0;
- *ancestor = 0;
- *tag = 0;
- loglen = 0;
-
- initial_commit = 0;
-}
-
-static void update_file(char *line)
-{
- char *name, *version;
- char *dir;
-
- while (isspace(*line))
- line++;
- name = line;
- line = strchr(line, ':');
- if (!line)
- return;
- *line++ = 0;
- line = strchr(line, '>');
- if (!line)
- return;
- *line++ = 0;
- version = line;
- line = strchr(line, '(');
- if (line) { /* "(DEAD)" */
- printf("git-update-cache --force-remove '%s'\n", name);
- return;
- }
-
- dir = strrchr(name, '/');
- if (dir)
- printf("mkdir -p %.*s\n", (int)(dir - name), name);
-
- printf("cvs -q -d %s checkout -d .git-tmp -r%s '%s/%s'\n",
- cvsroot, version, cvsmodule, name);
- printf("mv -f .git-tmp/%s %s\n", dir ? dir+1 : name, name);
- printf("rm -rf .git-tmp\n");
- printf("git-update-cache --add -- '%s'\n", name);
-}
-
-struct hdrentry {
- const char *name;
- char *dest;
-} hdrs[] = {
- { "Date:", date },
- { "Author:", author },
- { "Branch:", branch },
- { "Ancestor branch:", ancestor },
- { "Tag:", tag },
- { "Log:", NULL },
- { NULL, NULL }
-};
-
-int main(int argc, char **argv)
-{
- static char line[1000];
- enum state state = Header;
- int i;
-
- for (i = 1; i < argc; i++) {
- const char *arg = argv[i];
- if (!memcmp(arg, "--cvsroot=", 10)) {
- cvsroot = arg + 10;
- continue;
- }
- if (!memcmp(arg, "--module=", 9)) {
- cvsmodule = arg+9;
- continue;
- }
- if (!strcmp(arg, "-v")) {
- verbose = 1;
- continue;
- }
- if (!strcmp(arg, "-u")) {
- initial_commit = 0;
- continue;
- }
- }
-
-
- if (!cvsroot)
- cvsroot = getenv("CVSROOT");
-
- if (!cvsmodule || !cvsroot) {
- fprintf(stderr, "I need a CVSROOT and module name\n");
- exit(1);
- }
-
- if (initial_commit) {
- printf("[ -d .git ] && exit 1\n");
- printf("git-init-db\n");
- printf("mkdir -p .git/refs/heads\n");
- printf("mkdir -p .git/refs/tags\n");
- printf("ln -sf refs/heads/master .git/HEAD\n");
- }
-
- while (fgets(line, sizeof(line), stdin) != NULL) {
- int linelen = strlen(line);
-
- while (linelen && isspace(line[linelen-1]))
- line[--linelen] = 0;
-
- switch (state) {
- struct hdrentry *entry;
-
- case Header:
- if (verbose)
- printf("# H: %s\n", line);
- for (entry = hdrs ; entry->name ; entry++) {
- int len = strlen(entry->name);
- char *val;
-
- if (memcmp(entry->name, line, len))
- continue;
- if (!entry->dest) {
- state = Log;
- break;
- }
- val = line + len;
- linelen -= len;
- while (isspace(*val)) {
- val++;
- linelen--;
- }
- memcpy(entry->dest, val, linelen+1);
- break;
- }
- continue;
-
- case Log:
- if (verbose)
- printf("# L: %s\n", line);
- if (!strcmp(line, "Members:")) {
- while (loglen && isspace(log[loglen-1]))
- log[--loglen] = 0;
- prepare_commit();
- state = Members;
- continue;
- }
-
- if (loglen + linelen + 5 > sizeof(log))
- continue;
- memcpy(log + loglen, line, linelen);
- loglen += linelen;
- log[loglen++] = '\n';
- continue;
-
- case Members:
- if (verbose)
- printf("# M: %s\n", line);
- if (!linelen) {
- commit();
- state = Header;
- continue;
- }
- update_file(line);
- continue;
- }
- }
- return 0;
-}
diff --git a/git-cvsimport-script b/git-cvsimport-script
--- a/git-cvsimport-script
+++ b/git-cvsimport-script
@@ -1,38 +1,629 @@
-#!/bin/sh
+#!/usr/bin/perl -w
-usage () {
- echo "Usage: git cvsimport [-v] [-z fuzz] <cvsroot> <module>"
- exit 1
-}
-
-CVS2GIT=""
-CVSPS="--cvs-direct -x -A"
-while true; do
- case "$1" in
- -v) CVS2GIT="$1" ;;
- -z) shift; CVSPS="$CVSPS -z $1" ;;
- -*) usage ;;
- *) break ;;
- esac
- shift
-done
-
-export CVSROOT="$1"
-export MODULE="$2"
-if [ ! "$CVSROOT" ] || [ ! "$MODULE" ] ; then
- usage
-fi
-
-cvsps -h 2>&1 | grep -q "cvsps version 2.1" >& /dev/null || {
- echo "I need cvsps version 2.1"
- exit 1
-}
-
-mkdir "$MODULE" || exit 1
-cd "$MODULE"
-
-TZ=UTC cvsps $CVSPS $MODULE > .git-cvsps-result
-[ -s .git-cvsps-result ] || exit 1
-git-cvs2git $CVS2GIT --cvsroot="$CVSROOT" --module="$MODULE" < .git-cvsps-result > .git-create-script || exit 1
-sh .git-create-script
+# This tool is copyright (c) 2005, Matthias Urlichs.
+# It is released under the Gnu Public License, version 2.
+#
+# The basic idea is to aggregate CVS check-ins into related changes.
+# Fortunately, "cvsps" does that for us; all we have to do is to parse
+# its output.
+#
+# Checking out the files is done by a single long-running CVS connection
+# / server process.
+#
+# The head revision is on branch "origin" by default.
+# You can change that with the '-o' option.
+
+use strict;
+use warnings;
+use Getopt::Std;
+use File::Path qw(mkpath);
+use File::Basename qw(basename dirname);
+use Time::Local;
+use IO::Socket;
+use IO::Pipe;
+use POSIX qw(strftime dup2);
+
+$SIG{'PIPE'}="IGNORE";
+$ENV{'TZ'}="UTC";
+
+our($opt_h,$opt_o,$opt_v,$opt_d);
+
+sub usage() {
+ print STDERR <<END;
+Usage: ${\basename $0} # fetch/update GIT from CVS
+ [ -o branch-for-HEAD ] [ -h ] [ -v ] [ -d CVSROOT ]
+ CVS_module [ GIT_repository ]
+END
+ exit(1);
+}
+
+getopts("hqvo:d:") or usage();
+usage if $opt_h;
+
+@ARGV == 1 or @ARGV == 2 or usage();
+
+my($cvs_tree, $git_tree) = @ARGV;
+
+if($opt_d) {
+ $ENV{"CVSROOT"} = $opt_d;
+} elsif($ENV{"CVSROOT"}) {
+ $opt_d = $ENV{"CVSROOT"};
+} else {
+ die "CVSROOT needs to be set";
+}
+$opt_o ||= "origin";
+$git_tree ||= ".";
+
+select(STDERR); $|=1; select(STDOUT);
+
+
+package CVSconn;
+# Basic CVS dialog.
+# We're only interested in connecting and downloading, so ...
+
+use POSIX qw(strftime dup2);
+
+sub new {
+ my($what,$repo,$subdir) = @_;
+ $what=ref($what) if ref($what);
+
+ my $self = {};
+ $self->{'buffer'} = "";
+ bless($self,$what);
+
+ $repo =~ s#/+$##;
+ $self->{'fullrep'} = $repo;
+ $self->conn();
+
+ $self->{'subdir'} = $subdir;
+ $self->{'lines'} = undef;
+
+ return $self;
+}
+
+sub conn {
+ my $self = shift;
+ my $repo = $self->{'fullrep'};
+ if($repo =~ s/^:pserver:(?:(.*?)(?::(.*?))?@)?([^:\/]*)(?::(\d*))?//) {
+ my($user,$pass,$serv,$port) = ($1,$2,$3,$4);
+ $user="anonymous" unless defined $user;
+ my $rr2 = "-";
+ unless($port) {
+ $rr2 = ":pserver:$user\@$serv:$repo";
+ $port=2401;
+ }
+ my $rr = ":pserver:$user\@$serv:$port$repo";
+
+ unless($pass) {
+ open(H,$ENV{'HOME'}."/.cvspass") and do {
+ # :pserver:cvs@mea.tmt.tele.fi:/cvsroot/zmailer Ah<Z
+ while(<H>) {
+ chomp;
+ s/^\/\d+\s+//;
+ my ($w,$p) = split(/\s/,$_,2);
+ if($w eq $rr or $w eq $rr2) {
+ $pass = $p;
+ last;
+ }
+ }
+ };
+ }
+ $pass="A" unless $pass;
+
+ my $s = IO::Socket::INET->new(PeerHost => $serv, PeerPort => $port);
+ die "Socket to $serv: $!\n" unless defined $s;
+ $s->write("BEGIN AUTH REQUEST\n$repo\n$user\n$pass\nEND AUTH REQUEST\n")
+ or die "Write to $serv: $!\n";
+ $s->flush();
+
+ my $rep = <$s>;
+
+ if($rep ne "I LOVE YOU\n") {
+ $rep="<unknown>" unless $rep;
+ die "AuthReply: $rep\n";
+ }
+ $self->{'socketo'} = $s;
+ $self->{'socketi'} = $s;
+ } else { # local: Fork off our own cvs server.
+ my $pr = IO::Pipe->new();
+ my $pw = IO::Pipe->new();
+ my $pid = fork();
+ die "Fork: $!\n" unless defined $pid;
+ unless($pid) {
+ $pr->writer();
+ $pw->reader();
+ dup2($pw->fileno(),0);
+ dup2($pr->fileno(),1);
+ $pr->close();
+ $pw->close();
+ exec("cvs","server");
+ }
+ $pw->writer();
+ $pr->reader();
+ $self->{'socketo'} = $pw;
+ $self->{'socketi'} = $pr;
+ }
+ $self->{'socketo'}->write("Root $repo\n");
+
+ # Trial and error says that this probably is the minimum set
+ $self->{'socketo'}->write("Valid-responses ok error Valid-requests Mode M Mbinary E F Checked-in Created Updated Merged Removed\n");
+
+ $self->{'socketo'}->write("valid-requests\n");
+ $self->{'socketo'}->flush();
+
+ chomp(my $rep=$self->readline());
+ if($rep !~ s/^Valid-requests\s*//) {
+ $rep="<unknown>" unless $rep;
+ die "Expected Valid-requests from server, but got: $rep\n";
+ }
+ chomp(my $res=$self->readline());
+ die "validReply: $res\n" if $res ne "ok";
+
+ $self->{'socketo'}->write("UseUnchanged\n") if $rep =~ /\bUseUnchanged\b/;
+ $self->{'repo'} = $repo;
+}
+
+sub readline {
+ my($self) = @_;
+ return $self->{'socketi'}->getline();
+}
+
+sub _file {
+ # Request a file with a given revision.
+ # Trial and error says this is a good way to do it. :-/
+ my($self,$fn,$rev) = @_;
+ $self->{'socketo'}->write("Argument -N\n") or return undef;
+ $self->{'socketo'}->write("Argument -P\n") or return undef;
+ # $self->{'socketo'}->write("Argument -ko\n") or return undef;
+ # -ko: Linus' version doesn't use it
+ $self->{'socketo'}->write("Argument -r\n") or return undef;
+ $self->{'socketo'}->write("Argument $rev\n") or return undef;
+ $self->{'socketo'}->write("Argument --\n") or return undef;
+ $self->{'socketo'}->write("Argument $self->{'subdir'}/$fn\n") or return undef;
+ $self->{'socketo'}->write("Directory .\n") or return undef;
+ $self->{'socketo'}->write("$self->{'repo'}\n") or return undef;
+ $self->{'socketo'}->write("Sticky T1.1\n") or return undef;
+ $self->{'socketo'}->write("co\n") or return undef;
+ $self->{'socketo'}->flush() or return undef;
+ $self->{'lines'} = 0;
+ return 1;
+}
+sub _line {
+ # Read a line from the server.
+ # ... except that 'line' may be an entire file. ;-)
+ my($self) = @_;
+ die "Not in lines" unless defined $self->{'lines'};
+
+ my $line;
+ my $res="";
+ while(defined($line = $self->readline())) {
+ # M U gnupg-cvs-rep/AUTHORS
+ # Updated gnupg-cvs-rep/
+ # /daten/src/rsync/gnupg-cvs-rep/AUTHORS
+ # /AUTHORS/1.1///T1.1
+ # u=rw,g=rw,o=rw
+ # 0
+ # ok
+
+ if($line =~ s/^(?:Created|Updated) //) {
+ $line = $self->readline(); # path
+ $line = $self->readline(); # Entries line
+ my $mode = $self->readline(); chomp $mode;
+ $self->{'mode'} = $mode;
+ defined (my $cnt = $self->readline())
+ or die "EOF from server after 'Changed'\n";
+ chomp $cnt;
+ die "Duh: Filesize $cnt" if $cnt !~ /^\d+$/;
+ $line="";
+ $res="";
+ while($cnt) {
+ my $buf;
+ my $num = $self->{'socketi'}->read($buf,$cnt);
+ die "Server: Filesize $cnt: $num: $!\n" if not defined $num or $num<=0;
+ $res .= $buf;
+ $cnt -= $num;
+ }
+ } elsif($line =~ s/^ //) {
+ $res .= $line;
+ } elsif($line =~ /^M\b/) {
+ # output, do nothing
+ } elsif($line =~ /^Mbinary\b/) {
+ my $cnt;
+ die "EOF from server after 'Mbinary'" unless defined ($cnt = $self->readline());
+ chomp $cnt;
+ die "Duh: Mbinary $cnt" if $cnt !~ /^\d+$/ or $cnt<1;
+ $line="";
+ while($cnt) {
+ my $buf;
+ my $num = $self->{'socketi'}->read($buf,$cnt);
+ die "S: Mbinary $cnt: $num: $!\n" if not defined $num or $num<=0;
+ $res .= $buf;
+ $cnt -= $num;
+ }
+ } else {
+ chomp $line;
+ if($line eq "ok") {
+ # print STDERR "S: ok (".length($res).")\n";
+ return $res;
+ } elsif($line =~ s/^E //) {
+ # print STDERR "S: $line\n";
+ } else {
+ die "Unknown: $line\n";
+ }
+ }
+ }
+}
+sub file {
+ my($self,$fn,$rev) = @_;
+ my $res;
+
+ if ($self->_file($fn,$rev)) {
+ $res = $self->_line();
+ return $res if defined $res;
+ }
+
+ # retry
+ $self->conn();
+ $self->_file($fn,$rev)
+ or die "No file command send\n";
+ $res = $self->_line();
+ die "No input: $fn $rev\n" unless defined $res;
+ return $res;
+}
+
+
+package main;
+
+my $cvs = CVSconn->new($opt_d, $cvs_tree);
+
+
+sub pdate($) {
+ my($d) = @_;
+ m#(\d{2,4})/(\d\d)/(\d\d)\s(\d\d):(\d\d)(?::(\d\d))?#
+ or die "Unparseable date: $d\n";
+ my $y=$1; $y-=1900 if $y>1900;
+ return timegm($6||0,$5,$4,$3,$2-1,$y);
+}
+
+sub pmode($) {
+ my($mode) = @_;
+ my $m = 0;
+ my $mm = 0;
+ my $um = 0;
+ for my $x(split(//,$mode)) {
+ if($x eq ",") {
+ $m |= $mm&$um;
+ $mm = 0;
+ $um = 0;
+ } elsif($x eq "u") { $um |= 0700;
+ } elsif($x eq "g") { $um |= 0070;
+ } elsif($x eq "o") { $um |= 0007;
+ } elsif($x eq "r") { $mm |= 0444;
+ } elsif($x eq "w") { $mm |= 0222;
+ } elsif($x eq "x") { $mm |= 0111;
+ } elsif($x eq "=") { # do nothing
+ } else { die "Unknown mode: $mode\n";
+ }
+ }
+ $m |= $mm&$um;
+ return $m;
+}
+
+my $tmpcv = "/var/cache/cvs";
+
+sub getwd() {
+ my $pwd = `pwd`;
+ chomp $pwd;
+ return $pwd;
+}
+
+-d $git_tree
+ or mkdir($git_tree,0777)
+ or die "Could not create $git_tree: $!";
+chdir($git_tree);
+
+my $last_branch = "";
+my $orig_branch = "";
+my %branch_date;
+
+my $git_dir = $ENV{"GIT_DIR"} || ".git";
+$git_dir = getwd()."/".$git_dir unless $git_dir =~ m#^/#;
+$ENV{"GIT_DIR"} = $git_dir;
+unless(-d $git_dir) {
+ system("git-init-db");
+ die "Cannot init the GIT db at $git_tree: $?\n" if $?;
+ system("git-read-tree");
+ die "Cannot init an empty tree: $?\n" if $?;
+
+ $last_branch = $opt_o;
+ $orig_branch = "";
+} else {
+ $last_branch = basename(readlink("$git_dir/HEAD"));
+ unless($last_branch) {
+ warn "Cannot read the last branch name: $! -- assuming 'master'\n";
+ $last_branch = "master";
+ }
+ $orig_branch = $last_branch;
+
+ # Get the last import timestamps
+ opendir(D,"$git_dir/refs/heads");
+ while(defined(my $head = readdir(D))) {
+ next if $head =~ /^\./;
+ open(F,"$git_dir/refs/heads/$head")
+ or die "Bad head branch: $head: $!\n";
+ chomp(my $ftag = <F>);
+ close(F);
+ open(F,"git-cat-file commit $ftag |");
+ while(<F>) {
+ next unless /^author\s.*\s(\d+)\s[-+]\d{4}$/;
+ $branch_date{$head} = $1;
+ last;
+ }
+ close(F);
+ }
+ closedir(D);
+}
+
+-d $git_dir
+ or die "Could not create git subdir ($git_dir).\n";
+
+my $pid = open(CVS,"-|");
+die "Cannot fork: $!\n" unless defined $pid;
+unless($pid) {
+ exec("cvsps","-A","--cvs-direct",$cvs_tree);
+ die "Could not start cvsps: $!\n";
+}
+
+
+## cvsps output:
+#---------------------
+#PatchSet 314
+#Date: 1999/09/18 13:03:59
+#Author: wkoch
+#Branch: STABLE-BRANCH-1-0
+#Ancestor branch: HEAD
+#Tag: (none)
+#Log:
+# See ChangeLog: Sat Sep 18 13:03:28 CEST 1999 Werner Koch
+#Members:
+# README:1.57->1.57.2.1
+# VERSION:1.96->1.96.2.1
+#
+#---------------------
+
+my $state = 0;
+
+my($patchset,$date,$author,$branch,$ancestor,$tag,$logmsg);
+my(@old,@new);
+my $commit = sub {
+ my $pid;
+ system("git-update-cache","--force-remove","--",@old) if @old;
+ die "Cannot remove files: $?\n" if $?;
+ system("git-update-cache","--add","--",@new) if @new;
+ die "Cannot add files: $?\n" if $?;
+
+ $pid = open(C,"-|");
+ die "Cannot fork: $!" unless defined $pid;
+ unless($pid) {
+ exec("git-write-tree");
+ die "Cannot exec git-write-tree: $!\n";
+ }
+ chomp(my $tree = <C>);
+ length($tree) == 40
+ or die "Cannot get tree id ($tree): $!\n";
+ close(C)
+ or die "Error running git-write-tree: $?\n";
+ print "Tree ID $tree\n" if $opt_v;
+
+ my $parent = "";
+ if(open(C,"$git_dir/refs/heads/$last_branch")) {
+ chomp($parent = <C>);
+ close(C);
+ length($parent) == 40
+ or die "Cannot get parent id ($parent): $!\n";
+ print "Parent ID $parent\n" if $opt_v;
+ }
+
+ my $pr = IO::Pipe->new();
+ my $pw = IO::Pipe->new();
+ $pid = fork();
+ die "Fork: $!\n" unless defined $pid;
+ unless($pid) {
+ $pr->writer();
+ $pw->reader();
+ dup2($pw->fileno(),0);
+ dup2($pr->fileno(),1);
+ $pr->close();
+ $pw->close();
+
+ my @par = ();
+ @par = ("-p",$parent) if $parent;
+ exec("env",
+ "GIT_AUTHOR_NAME=$author",
+ "GIT_AUTHOR_EMAIL=$author",
+ "GIT_AUTHOR_DATE=".strftime("+0000 %Y-%m-%d %H:%M:%S",gmtime($date)),
+ "GIT_COMMITTER_NAME=$author",
+ "GIT_COMMITTER_EMAIL=$author",
+ "GIT_COMMITTER_DATE=".strftime("+0000 %Y-%m-%d %H:%M:%S",gmtime($date)),
+ "git-commit-tree", $tree,@par);
+ die "Cannot exec git-commit-tree: $!\n";
+ }
+ $pw->writer();
+ $pr->reader();
+ print $pw $logmsg
+ or die "Error writing to git-commit-tree: $!\n";
+ $pw->close();
+
+ print "Committed patch $patchset ($branch)\n" if $opt_v;
+ chomp(my $cid = <$pr>);
+ length($cid) == 40
+ or die "Cannot get commit id ($cid): $!\n";
+ print "Commit ID $cid\n" if $opt_v;
+ $pr->close();
+
+ waitpid($pid,0);
+ die "Error running git-commit-tree: $?\n" if $?;
+
+ open(C,">$git_dir/refs/heads/$branch")
+ or die "Cannot open branch $branch for update: $!\n";
+ print C "$cid\n"
+ or die "Cannot write branch $branch for update: $!\n";
+ close(C)
+ or die "Cannot write branch $branch for update: $!\n";
+
+ if($tag) {
+ open(C,">$git_dir/refs/tags/$tag")
+ or die "Cannot create tag $tag: $!\n";
+ print C "$cid\n"
+ or die "Cannot write tag $branch: $!\n";
+ close(C)
+ or die "Cannot write tag $branch: $!\n";
+ print "Created tag '$tag' on '$branch'\n" if $opt_v;
+ }
+
+ @old = ();
+ @new = ();
+};
+
+while(<CVS>) {
+ chomp;
+ if($state == 0 and /^-+$/) {
+ $state = 1;
+ } elsif($state == 0) {
+ $state = 1;
+ redo;
+ } elsif(($state==0 or $state==1) and s/^PatchSet\s+//) {
+ $patchset = 0+$_;
+ $state=2;
+ } elsif($state == 2 and s/^Date:\s+//) {
+ $date = pdate($_);
+ unless($date) {
+ print STDERR "Could not parse date: $_\n";
+ $state=0;
+ next;
+ }
+ $state=3;
+ } elsif($state == 3 and s/^Author:\s+//) {
+ s/\s+$//;
+ $author = $_;
+ $state = 4;
+ } elsif($state == 4 and s/^Branch:\s+//) {
+ s/\s+$//;
+ $branch = $_;
+ $state = 5;
+ } elsif($state == 5 and s/^Ancestor branch:\s+//) {
+ s/\s+$//;
+ $ancestor = $_;
+ $ancestor = $opt_o if $ancestor == "HEAD";
+ $state = 6;
+ } elsif($state == 5) {
+ $ancestor = undef;
+ $state = 6;
+ redo;
+ } elsif($state == 6 and s/^Tag:\s+//) {
+ s/\s+$//;
+ if($_ eq "(none)") {
+ $tag = undef;
+ } else {
+ $tag = $_;
+ }
+ $state = 7;
+ } elsif($state == 7 and /^Log:/) {
+ $logmsg = "";
+ $state = 8;
+ } elsif($state == 8 and /^Members:/) {
+ $branch = $opt_o if $branch eq "HEAD";
+ if(defined $branch_date{$branch} and $branch_date{$branch} >= $date) {
+ # skip
+ print "skip patchset $patchset: $date before $branch_date{$branch}\n";
+ $state = 11;
+ next;
+ }
+ if($ancestor) {
+ if(-f "$git_dir/refs/heads/$branch") {
+ print STDERR "Branch $branch already exists!\n";
+ $state=11;
+ next;
+ }
+ unless(open(H,"$git_dir/refs/heads/$ancestor")) {
+ print STDERR "Branch $ancestor does not exist!\n";
+ $state=11;
+ next;
+ }
+ chomp(my $id = <H>);
+ close(H);
+ unless(open(H,"> $git_dir/refs/heads/$branch")) {
+ print STDERR "Could not create branch $branch: $!\n";
+ $state=11;
+ next;
+ }
+ print H "$id\n"
+ or die "Could not write branch $branch: $!";
+ close(H)
+ or die "Could not write branch $branch: $!";
+ }
+ if(($ancestor || $branch) ne $last_branch) {
+ print "Switching from $last_branch to $branch\n" if $opt_v;
+ system("git-read-tree","-m","-u","$last_branch","$branch");
+ die "read-tree failed: $?\n" if $?;
+ }
+ if($branch ne $last_branch) {
+ unlink("$git_dir/HEAD");
+ symlink("refs/heads/$branch","$git_dir/HEAD");
+ $last_branch = $branch;
+ }
+ $state = 9;
+ } elsif($state == 8) {
+ $logmsg .= "$_\n";
+ } elsif($state == 9 and /^\s+(\S+):(INITIAL|\d(?:\.\d+)+)->(\d(?:\.\d+)+)\s*$/) {
+# VERSION:1.96->1.96.2.1
+ my $init = ($2 eq "INITIAL");
+ my $fn = $1;
+ my $rev = $3;
+ $fn =~ s#^/+##;
+ my $data = $cvs->file($fn,$rev);
+ print "".($init ? "New" : "Update")." $fn: ".length($data)." bytes.\n";
+ mkpath(dirname($fn),$opt_v);
+ open(F,"> ./$fn")
+ or die "Cannot create '$fn': $!\n";
+ print F $data
+ or die "Cannot write to '$fn': $!\n";
+ close(F)
+ or die "Cannot write to '$fn': $!\n";
+ chmod(pmode($cvs->{'mode'}), $fn);
+ push(@new,$fn); # may be resurrected!
+ } elsif($state == 9 and /^\s+(\S+):\d(?:\.\d+)+->(\d(?:\.\d+)+)\(DEAD\)\s*$/) {
+ my $fn = $1;
+ $fn =~ s#^/+##;
+ push(@old,$fn);
+ } elsif($state == 9 and /^\s*$/) {
+ $state = 10;
+ } elsif(($state == 9 or $state == 10) and /^-+$/) {
+ &$commit();
+ $state = 1;
+ } elsif($state == 11 and /^-+$/) {
+ $state = 1;
+ } elsif(/^-+$/) { # end of unknown-line processing
+ $state = 1;
+ } elsif($state != 11) { # ignore stuff when skipping
+ print "* UNKNOWN LINE * $_\n";
+ }
+}
+&$commit() if $branch and $state != 11;
+
+# Now switch back to the branch we were in before all of this happened
+if($orig_branch) {
+ print "DONE; switching back to $orig_branch\n" if $opt_v;
+} else {
+ $orig_branch = "master";
+ print "DONE; creating $orig_branch branch\n" if $opt_v;
+ system("cp","$git_dir/refs/heads/$opt_o","$git_dir/refs/heads/master")
+ unless -f "$git_dir/refs/heads/master";
+}
+
+system("git-read-tree","-m","-u","$last_branch","$orig_branch");
+die "read-tree failed: $?\n" if $?;
+
+unlink("$git_dir/HEAD");
+symlink("refs/heads/$orig_branch","$git_dir/HEAD");
^ permalink raw reply
* Re: GIT-CVS sync script
From: Matthias Urlichs @ 2005-06-28 19:33 UTC (permalink / raw)
To: git
In-Reply-To: <20050628191445.GA27979@lumumba.uhasselt.be>
Hi, Panagiotis Issaris wrote:
> I find the included script useful to easily track a CVS repository in
> GIT. You can both call it for the initial import and ofcourse for the
> subsequent resynchronisation with the parent CVS repository (that's what
> the script's about).
>
Hmm, my Perl script (just posted...) should do the same thing without
depending on .git* files which may or may not still exist when you do an
incremental import half a year later.
Can you test whether that script works for you?
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
- -
I may not be totally perfect, but parts of me are excellent.
-- Ashleigh Brilliant
^ permalink raw reply
* Re: kernel.org and GIT tree rebuilding
From: Linus Torvalds @ 2005-06-28 19:28 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Git Mailing List, Chris Mason
In-Reply-To: <Pine.LNX.4.63.0506281351150.1667@localhost.localdomain>
On Tue, 28 Jun 2005, Nicolas Pitre wrote:
>
> Here's one improvement to the pack format, breaking it early so it won't
> affect anyone at this point: compressed object header. Instead of the
> fixed 5 byte header, this patch convert it to a variable size granted
> most object are small enough to save on the storage of the significant
> size bytes which will be zero and packing the non-zero byte position
> with the object type.
Ok, this is against an older version and doesn't have that "read_sha1"
thing, but yes, something like this would work.
I'd prefer the encoding to be a bit different, though: make the size be
encoded in seven bits per byte, with the high bit meaning "more to come".
We can use four bits from the "type" byte for the initial value, making
lengths 0-15 be free.
unsigned long size;
unsigned char c;
c = *pack++;
type =
size = c & 15;
type = (c >> 4) & 7;
while (c & 0x80) {
c = *pack++;
size = (size << 7) + (pack & 0x7f);
}
or something. That's even denser.
However, I also end up wanting to add a "global header" to the pack-file,
that contains at least the number of objects packed. We may not know how
big the pack-file will be, but we'll at least know how many objects it
has before we start writing it.
Linus
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Matthias Urlichs @ 2005-06-28 19:49 UTC (permalink / raw)
To: git
In-Reply-To: <Pine.LNX.4.58.0506281111480.19755@ppc970.osdl.org>
Hi, Linus Torvalds wrote:
> Ugly, in any case.
Why not chunk the thing?
In other words, the stream shouldn't be
"here's a big-ass packfile of unknown size"
but an arbitrary number of
"here's a N-byte sized chunk of the current pack file"
snippets, followed by a
"here's the SHA1 of the whole thing"
packet.
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
- -
Be like a duck -- keep calm and unruffled on the surface but paddle like the
devil under water.
^ permalink raw reply
* Re: GIT-CVS sync script
From: Linus Torvalds @ 2005-06-28 19:56 UTC (permalink / raw)
To: Matthias Urlichs, Panagiotis Issaris; +Cc: Git Mailing List
In-Reply-To: <pan.2005.06.28.19.33.40.876796@smurf.noris.de>
On Tue, 28 Jun 2005, Matthias Urlichs wrote:
> Hi, Panagiotis Issaris wrote:
>
> > I find the included script useful to easily track a CVS repository in
> > GIT. You can both call it for the initial import and ofcourse for the
> > subsequent resynchronisation with the parent CVS repository (that's what
> > the script's about).
> >
> Hmm, my Perl script (just posted...) should do the same thing without
> depending on .git* files which may or may not still exist when you do an
> incremental import half a year later.
>
> Can you test whether that script works for you?
Please do. I'm not proud of my horrid cvs2git thing, and if just one other
user reports "it works for me" I'll happily replace that piece of cr*p
with Matthias' version that looks much nicer (but not being a perl user I
can't judge it any other way than having others say "yeah, that works for
me").
Linus
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Matthias Urlichs @ 2005-06-28 20:18 UTC (permalink / raw)
To: git
In-Reply-To: <pan.2005.06.28.19.49.33.828202@smurf.noris.de>
I wrote:
> Linus Torvalds wrote:
>
>> Ugly, in any case.
>
> Why not chunk the thing?
Having the number of files sent first would work too, I'd think.
I'm wary of trying to interpret something non-decompressible as a sha1
chunk, however -- the set of random bytes that, to zlib, look like a
sufficiently valid zip header that it wants to read more than 20 of them
before punting is certainly not zero.
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
- -
I was sure the old fellow would never make it
to the other side of the curb when I struck him.
^ permalink raw reply
* Re: new features in gitk
From: Luc Van Oostenryck @ 2005-06-28 20:24 UTC (permalink / raw)
To: Paul Mackerras; +Cc: git
In-Reply-To: <17088.36232.479492.643878@cargo.ozlabs.ibm.com>
Paul Mackerras wrote:
> Luc Van Oostenryck writes:
>
>
>>I find the patch generation very usefull (when I've seen the tool a few days ago, I've said to myself:
>>"what a wonderfull tool, if only I could create a patch from this")
>>but it doesn't work for me (the first three entries stay always in grey, only the last one "Create tag" is in black).
>>I'm missing something ?
>
>
> The diffs and patches are between the selected commit (one that you
> clicked on with the left button and which is shown with a grey
> background) and the commit where you click the right button. That was
> the best way I could think of to indicate which were the two commits
> to diff. If you (or anyone) has a better suggestion, let me know.
>
> Paul.
OK I see it now, it works nicely.
I didn't found the tric yesterday because I was thinking: one commit -> one patch,
but it's probably more usefull like it is now.
Thanks.
^ permalink raw reply
* Re: Mercurial vs Updated git HOWTO for kernel hackers
From: Kyle Moffett @ 2005-06-28 20:27 UTC (permalink / raw)
To: Matt Mackall
Cc: mercurial, Petr Baudis, Linux Kernel, Jeff Garzik,
Git Mailing List
In-Reply-To: <20050628180157.GI12006@waste.org>
On Jun 28, 2005, at 14:01:57, Matt Mackall wrote:
> Everything in Mercurial is an append-only log. A transaction journal
> records the original length of each log so that it can be restored on
> failure.
Does this mean that (excepting the "undo" feature) one could set the
ext3 "append-only" attribute on the repository files to avoid losing
data due to user account compromise?
Cheers,
Kyle Moffett
--
I lost interest in "blade servers" when I found they didn't throw
knives at people who weren't supposed to be in your machine room.
-- Anthony de Boer
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Petr Baudis @ 2005-06-28 20:30 UTC (permalink / raw)
To: Benjamin LaHaise; +Cc: Christopher Li, Junio C Hamano, Linus Torvalds, git
In-Reply-To: <20050628163551.GA29410@kvack.org>
Dear diary, on Tue, Jun 28, 2005 at 06:35:51PM CEST, I got a letter
where Benjamin LaHaise <bcrl@kvack.org> told me that...
> On Tue, Jun 28, 2005 at 04:52:56PM +0200, Petr Baudis wrote:
> > I think the git-*-pull tools are actually just fine. You will only need
> > to have some server-side CGI gadget to frontend the file, but we need
> > that anyway to make the pull reasonably effective.
>
> Not really -- the use of rsync for the objects fails horribly on slow
> links when the project scales in the number of commits. The rsync
> protocol has to transfer the names of each file and some information
> about it, and that information isn't delta compressed. This is where
> kernel.org is falling over, as well as what makes the kernel tree very
> painful to use over a dialup modem link.
Yes. But isn't that what I'm after all saying too? git-*-pull tools
shouldn't have that problem since they have much less overhead and only
pull stuff you need.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
<Espy> be careful, some twit might quote you out of context..
^ permalink raw reply
* Re: Mercurial vs Updated git HOWTO for kernel hackers
From: Sean @ 2005-06-28 20:45 UTC (permalink / raw)
To: Kyle Moffett
Cc: Matt Mackall, Petr Baudis, Christopher Li, Jeff Garzik,
Linux Kernel, Git Mailing List, mercurial
In-Reply-To: <62CF578B-B9DF-4DEA-8BAD-041F357771FD@mac.com>
On Tue, June 28, 2005 4:27 pm, Kyle Moffett said:
> On Jun 28, 2005, at 14:01:57, Matt Mackall wrote:
>> Everything in Mercurial is an append-only log. A transaction journal
>> records the original length of each log so that it can be restored on
>> failure.
>
> Does this mean that (excepting the "undo" feature) one could set the
> ext3 "append-only" attribute on the repository files to avoid losing
> data due to user account compromise?
>
Probably. In Git, which is a bit more flexible than Mecurial you can
chmod your objects to read-only or use the ext3 immutable setting to
protect your existing objects. You can even have a setup where objects
are archived onto write-once media like DVD and still participate in a
live repository, where new objects are written to hard disk, but older
object are (automatically) sourced from the DVD.
Sean
^ permalink raw reply
* Re: Cogito vs. Git (was: Mercurial vs Updated git HOWTO for kernel hackers)
From: Petr Baudis @ 2005-06-28 20:54 UTC (permalink / raw)
To: Kevin Smith; +Cc: Git Mailing List
In-Reply-To: <42C1800C.7040506@qualitycode.com>
Dear diary, on Tue, Jun 28, 2005 at 06:51:24PM CEST, I got a letter
where Kevin Smith <yarcs@qualitycode.com> told me that...
> Petr Baudis wrote:
> >Cogito's only unusual requirement (well, expectation) is that HEAD is a
> >symlink to .git/refs/heads/master, and .git/refs/heads/master should
> >reflect your current head. I will try to ease up this restriction so
> >that things will mostly work even if you just have HEAD. I think that
> >most auxiliary commands (e.g. cg-log - you just have to love it) should
> >work on any sensible git tree (but I didn't test it - yet).
>
> So you're saying that aside from that one solveable issue, I could use
> low-level git tools, third-party over-git tools, and cogito,
> interchangebly on a single repo without the tools becoming confused?
Yes, given that you use the toolset "atomically" - if you start doing
some large-scale operation using one tool, you should finish it using
the same tool. That is, if you start merging by Cogito and then try to
commit by Linus' git plumbing, it won't work (and vice versa). cg-seek
combined with third-party over-git tools might have unknown consequences
as well.
> That's cool, and should be made more clear in the cogito readme. I was
> under the impression that cogito was tracking all kinds of extra meta
> magic stuff that git tools wouldn't keep updated.
It is tracking only seeks and merges in way the git tools wouldn't keep
updated. I will try to actually make it use ~/.git/MERGE_HEAD instead of
~/.git/merging as well, to make people's live even easier.
> If I were using cogito, I probably wouldn't want to use the low-level
> git stuff directly, but I might want to use (or maybe even write) some
> other over-git tools.
>
> You might also consider removing the "Core GIT" section from the README,
> because I think it increases the confusion between the two.
I might consider throwing away Git from Cogito distribution altogether,
but that would require Git to have some actual release cycle. ;-)
FYI, I will merge the pile of queued Cogito
patches during the weekend and next week, preparing 0.11.4 or 0.12
(depends on how I feel about it). I will also merge the Linus' stuff,
but only up to:
7323aa11af1527d5a786d93ee34401c72c5df051
Jan Harkes 2005-06-25 13:41
[PATCH] git-write-tree doesn't check alternate directories
The next patch is
c323ac7d9c573c5ee8b45b9b9def92a4d4d8204d
Linus Torvalds 2005-06-25 14:42
git-pack-objects: create a packed object representation
and I'm not sure if I'm ready to already take the packed stuff; I will
see how tested it gets (and if fsck will get taught about it etc). If it
won't be ready by the time I'm ready for the release, I will merge the
rest right after the release.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
<Espy> be careful, some twit might quote you out of context..
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Daniel Barkalow @ 2005-06-28 20:01 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.58.0506281111480.19755@ppc970.osdl.org>
On Tue, 28 Jun 2005, Linus Torvalds wrote:
> On Tue, 28 Jun 2005, Linus Torvalds wrote:
> >
> > I can certainly add an option to git-pack-file that disables writing of
> > the index file, and just writes the pack-file to stdout.
>
> Done.
What I actually meant was that it would be useful for git-ssh-push to be
able to pack stuff as a function call rather than execing an external
program, because just sticking git-ssh-push at the end of a pipeline
doesn't work if you don't remember what the remote side has.
> > I'm not sure I
> > want to write the "parse incoming pack-file" thing, but git-unpack-objects
> > comes _reasonably_ close (but right now it seeks around using the index
> > file to resolve deltas, instead of keeping them in memory and resolving
> > them when possible).
>
> I'm still thinking about this one. I think I'll just do it.
One possibility would be to put a special type tag (like '\0') before the
hash, so that the format is more deterministic.
> One problem here is that since we don't know how big the incoming
> pack-file will be, in a streaming input environment the receiver needs to
> either make the pack-file reception be the last thing it sees, or it will
> have to live with the fact that "git-unpack-objects" will read some more
> than it needs before it notices that it got it all...
In a completely streaming environment, yes; but the receiving side is the
one sending commands, so you don't run into the next thing unless you're
overlapping requests. Failing that, we can just keep a 4k buffer of stuff
we've already read around; we don't have to worry about reading into
something we won't want to read at all.
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox