* Re: Creating objects manually and repack
From: Martin Langhoff @ 2006-08-05 4:15 UTC (permalink / raw)
To: Jon Smirl; +Cc: Linus Torvalds, git
In-Reply-To: <9e4733910608041017v235da03ocd3eeeb0ba0e259b@mail.gmail.com>
On 8/5/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> On 8/4/06, Linus Torvalds <torvalds@osdl.org> wrote:
> > and you're basically all done. The above would turn each *,v file into a
> > *-<sha>.pack/*-<sha>.idx file pair, so you'd have exactly as many
> > pack-files as you have *,v files.
>
> I'll end up with 110,000 pack files.
Then just do it every 100 files, and you'll only have 1,100 pack
files, and it'll be fine.
> I suspect when I run repack over
> that it is going to take 24hrs or more,
Probably, but only the initial import has to incur that huge cost.
cheers,
martin
^ permalink raw reply
* Re: Creating objects manually and repack
From: Jon Smirl @ 2006-08-05 5:12 UTC (permalink / raw)
To: Martin Langhoff; +Cc: Linus Torvalds, git
In-Reply-To: <46a038f90608042115m71adc8ffo77de7940efa847a8@mail.gmail.com>
On 8/5/06, Martin Langhoff <martin.langhoff@gmail.com> wrote:
> On 8/5/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> > On 8/4/06, Linus Torvalds <torvalds@osdl.org> wrote:
> > > and you're basically all done. The above would turn each *,v file into a
> > > *-<sha>.pack/*-<sha>.idx file pair, so you'd have exactly as many
> > > pack-files as you have *,v files.
> >
> > I'll end up with 110,000 pack files.
>
> Then just do it every 100 files, and you'll only have 1,100 pack
> files, and it'll be fine.
This is something that has to be tuned. If you wait too long
everything spills out of RAM and you go totally IO bound for days. If
you do it too often you end up with too many packs and it takes a day
to repack them.
If I had a way to pipe the all of the objects into repack one at a
time without repack doing multiple passes none of this tuning would be
necessary. In this model the standalone objects never get created in
the first place. The fastest IO is IO that has been eliminated.
> > I suspect when I run repack over
> > that it is going to take 24hrs or more,
>
> Probably, but only the initial import has to incur that huge cost.
Mozilla developers aren't all rushing to switch to git. A switch needs
to be as painless as possible. If things are too complex they simply
won't switch.
Switching Mozilla to git is going to require a sales job and proof
that the tools are reliable and better than CVS. Right now I can't
even reliably import Mozilla CVS. One of the conditions for even
considering git is that they can easily do the CVS import internally
and verify it for accuracy.
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* [RFC/PATCH] Fix "grep -w"
From: Junio C Hamano @ 2006-08-05 5:16 UTC (permalink / raw)
To: git
We used to find the first match of the pattern and then if the
match is not for the entire word, declared that the whole line
does not match.
But that is wrong. The command "git grep -w -e mmap" should
find that a line "foo_mmap bar mmap baz" matches, by tring the
second instance of pattern "mmap" on the same line.
Signed-off-by: Junio C Hamano <junkio@cox.net>
---
builtin-grep.c | 10 ++++++++++
t/t7002-grep.sh | 45 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 55 insertions(+), 0 deletions(-)
diff --git a/builtin-grep.c b/builtin-grep.c
index 69b7c48..b5feda4 100644
--- a/builtin-grep.c
+++ b/builtin-grep.c
@@ -412,6 +412,7 @@ static int match_one_pattern(struct grep
int hit = 0;
regmatch_t pmatch[10];
+ again:
if (!opt->fixed) {
regex_t *exp = &p->regexp;
hit = !regexec(exp, bol, ARRAY_SIZE(pmatch),
@@ -438,6 +439,15 @@ static int match_one_pattern(struct grep
if (pmatch[0].rm_eo != (eol-bol) &&
word_char(bol[pmatch[0].rm_eo]))
hit = 0;
+
+ if (!hit && pmatch[0].rm_eo + bol < eol) {
+ /* there could be more than one match on the
+ * line, and the first match might not be
+ * strict word match. But later ones could be!
+ */
+ bol += pmatch[0].rm_eo;
+ goto again;
+ }
}
return hit;
}
diff --git a/t/t7002-grep.sh b/t/t7002-grep.sh
new file mode 100755
index 0000000..0a0e302
--- /dev/null
+++ b/t/t7002-grep.sh
@@ -0,0 +1,45 @@
+#!/bin/sh
+#
+# Copyright (c) 2006 Junio C Hamano
+#
+
+test_description='git grep -w
+'
+
+. ./test-lib.sh
+
+test_expect_success setup '
+ {
+ echo foo mmap bar
+ echo foo_mmap bar
+ echo foo_mmap bar mmap
+ echo foo mmap bar_mmap
+ echo foo_mmap bar mmap baz
+ } >file &&
+ git add file &&
+ git commit -m initial
+'
+
+test_expect_success 'grep -w HEAD' '
+ git grep -n -w -e mmap HEAD >actual &&
+ {
+ echo HEAD:file:1:foo mmap bar
+ echo HEAD:file:3:foo_mmap bar mmap
+ echo HEAD:file:4:foo mmap bar_mmap
+ echo HEAD:file:5:foo_mmap bar mmap baz
+ } >expected &&
+ diff expected actual
+'
+
+test_expect_success 'grep -w in working tree' '
+ git grep -n -w -e mmap >actual &&
+ {
+ echo file:1:foo mmap bar
+ echo file:3:foo_mmap bar mmap
+ echo file:4:foo mmap bar_mmap
+ echo file:5:foo_mmap bar mmap baz
+ } >expected &&
+ diff expected actual
+'
+
+test_done
^ permalink raw reply related
* Re: Creating objects manually and repack
From: Shawn Pearce @ 2006-08-05 5:21 UTC (permalink / raw)
To: Jon Smirl; +Cc: Martin Langhoff, Linus Torvalds, git
In-Reply-To: <9e4733910608042212p6bf56224ye0ecf3f06b2840cf@mail.gmail.com>
Jon Smirl <jonsmirl@gmail.com> wrote:
> On 8/5/06, Martin Langhoff <martin.langhoff@gmail.com> wrote:
> >On 8/5/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> >> On 8/4/06, Linus Torvalds <torvalds@osdl.org> wrote:
> >> > and you're basically all done. The above would turn each *,v file into
> >a
> >> > *-<sha>.pack/*-<sha>.idx file pair, so you'd have exactly as many
> >> > pack-files as you have *,v files.
> >>
> >> I'll end up with 110,000 pack files.
> >
> >Then just do it every 100 files, and you'll only have 1,100 pack
> >files, and it'll be fine.
>
> This is something that has to be tuned. If you wait too long
> everything spills out of RAM and you go totally IO bound for days. If
> you do it too often you end up with too many packs and it takes a day
> to repack them.
>
> If I had a way to pipe the all of the objects into repack one at a
> time without repack doing multiple passes none of this tuning would be
> necessary. In this model the standalone objects never get created in
> the first place. The fastest IO is IO that has been eliminated.
I'm almost done with what I'm calling `git-fast-import`. It takes
a stream of blobs on STDIN and writes the pack to a file, printing
SHA1s in hex format to STDOUT. The basic format for STDIN is a 4
byte length (native format) followed by that many bytes of blob data.
It prints the SHA1 for that blob to STDOUT, then waits for another
length.
It naively deltas each object against the prior object, thus it
would be best to feed it one ,v file at a time working from the most
recent revision back to the oldest revision. This works well for
an RCS file as that's the natural order to process the file in. :-)
When done you close STDIN and it'll rip through and update the pack
object count and the trailing checksum. This should let you pack
the entire repository in delta format using only two passes over the
data: one to write out the pack file and one to compute its checksum.
I'll post the code in a couple of hours.
--
Shawn.
^ permalink raw reply
* Your health, Neo-sumerian
From: Elena Denny @ 2006-08-05 11:12 UTC (permalink / raw)
To: git-commits-head-owner
Even if you have no erectin problems SOFT CIA6LIS
would help you to make BETTER SEPX MORE OFTEN!
and to bring unimagnable plesure to her.
Just disolve half a pil under your tongue
and get ready for action in 15 minutes.
The tests showed that the majority of men
after taking this medic ation were able to have
PERFECT ERMECTION during 36 hours!
VISIT US, AND GET OUR SPECIAL 70% DISCZOUNT OFER!
http://iouebp.pulsemix.net/?99786305
==========
Flock, do we?" Fletcher said, rather self-consciously. "Besides, if
INSTITUTE FOR EXTRATERRESTRIAL CULTURES
There are several reasons--and a great many more hypotheses-- for this
He's too modest to bring it up, but that was our most interesting adventure
"See here Jonathan " said his father not unkindly. "Winter isn't far
It was a lot simpler after that. I found the crack, and it was still
learned by anybody who wanted to discover it; that's got nothing to do
"Most wonderful." I started edging toward the john.
^ permalink raw reply
* Git files data formats documentation
From: A Large Angry SCM @ 2006-08-05 5:39 UTC (permalink / raw)
To: git
[-- Attachment #1: Type: text/plain, Size: 79 bytes --]
This information may be useful for reading and writing the various Git
files.
[-- Attachment #2: dataformats.txt --]
[-- Type: text/plain, Size: 14398 bytes --]
Git files data formats
======================
OBJECTS
-------
# The object ID, or "name", of an object is
# _sha-1_digest_( <OBJECT_HEADER> <object_CONTENTS> ).
<BLOB>
: _deflate_( <OBJECT_HEADER> <BLOB_CONTENTS> )
| <COMPACT_OBJECT_HEADER> _deflate_( <BLOB_CONTENTS> )
;
<BLOB_CONTENTS>
: <DATA>
;
<TREE>
: _deflate_( <OBJECT_HEADER> <TREE_CONTENTS> )
| <COMPACT_OBJECT_HEADER> _deflate_( <TREE_CONTENTS> )
;
<TREE_CONTENTS>
: <TREE_ENTRIES>
;
<TREE_ENTRIES>
# Tree entries are sorted by the byte sequence that comprises
# the entry name.
: ( <TREE_ENTRY> )*
;
<TREE_ENTRY>
# The type of the object referenced MUST be appropriate for
# the mode. Regular files and symbolic links reference a BLOB
# and directories reference a TREE.
: <OCTAL_MODE> <SP> <NAME> <NUL> <BINARY_OBJ_ID>
;
<COMMIT>
: _deflate_( <OBJECT_HEADER> <COMMIT_CONTENTS> )
| <COMPACT_OBJECT_HEADER> _deflate_( <COMMIT_CONTENTS> )
;
<COMMIT_CONTENTS>
: "tree" <SP> <HEX_OBJ_ID> <LF>
( "parent" <SP> <HEX_OBJ_ID> <LF> )*
"author" <SP>
<SAFE_NAME> <SP>
<LT> <SAFE_EMAIL> <GT> <SP>
<GIT_DATE> <LF>
"committer" <SP>
<SAFE_NAME> <SP>
<LT> <SAFE_EMAIL> <GT> <SP>
<GIT_DATE> <LF>
<LF>
<DATA>
;
<TAG>
: _deflate_( <OBJECT_HEADER> <TAG_CONTENTS> )
| <COMPACT_OBJECT_HEADER> _deflate_( <TAG_CONTENTS> )
;
<TAG_CONTENTS>
: "object" <SP> <HEX_OBJ_ID> <LF>
"type" <SP> <NONTAG_OBJ_TYPE> <LF>
"tag" <SP> <TAG_NAME> <LF>
<LF>
<DATA>
;
<OBJECT_HEADER>
: <OBJ_TYPE> <SP> <DECIMAL_LENGTH> <NUL>
;
<COMPACT_OBJECT_HEADER>
# The object type DELTA_ENCODED is not valid in a
# <COMPACT_OBJECT_HEADER>.
: <TYPE_AND_BASE128_SIZE>
;
<DATA>
# Uninterpreted sequence of bytes.
;
<OCTAL_MODE>
# Octal encoding, without prefix, of the file system object
# type and permission bits. The bit layout is according to the
# POSIX standard, with only regular files, directories, and
# symbolic links permitted. The actual permission bits are
# all zero except for regular files. The only permission bit
# of any consequence to Git is the owner executable bit. By
# default, the permission bits for files will be either 0644
# or 0755, depending on the owner executable bit.
;
<NAME>
# Sequence of bytes not containing the ASCII character byte
# value NUL (0x00).
;
<BINARY_OBJ_ID>
# The object ID of the referenced object.
;
<HEX_OBJ_ID>
# Hexidecimal encoding (lower case) of the <BINARY_OBJ_ID>.
;
<SAFE_NAME>
: <SAFE_STRING>
;
<SAFE_EMAIL>
: <SAFE_STRING>
;
<SAFE_STRING>
# A sequence of bytes not containing the ASCII character byte
# values NUL (0x00), LF (0x0a), '<' (0c3c), or '>' (0x3e).
#
# The sequence may not begin or end with any bytes with the
# following ASCII character byte values: SPACE (0x20),
# '.' (0x2e), ',' (0x2c), ':' (0x3a), ';' (0x3b), '<' (0x3c),
# '>' (0x3e), '"' (0x22), "'" (0x27).
;
<GIT_DATE>
: <SECONDS> <SP> <TZ_OFFSET>
;
<SECONDS>
# Base 10, ASCII encoding of the number of seconds since 12:00
# midnight January 1, 1970, UTC without accounting for leap
# seconds, and without leading zeros.
;
<TZ_OFFSET>
# Signed offset of time zone from UTC.
: <TZ_OFFSET_SIGN> <TZ_OFFSET_HOURS> <TZ_OFFSET_MIN>
;
<TZ_OFFSET_SIGN>
: "+"
| "-"
;
<TZ_OFFSET_HOURS>
: <DIGIT> <DIGIT>
;
<TZ_OFFSET_MIN>
# Valid values are "00" to "59" inclusive.
: <DIGIT> <DIGIT>
;
<DIGIT>
# ASCII decimal digit.
;
<NONTAG_OBJ_TYPE>
: "BLOB"
| "TREE"
| "COMMIT"
;
<OBJ_TYPE>
: <NONTAG_OBJ_TYPE>
| "TAG"
;
<DECIMAL_LENGTH>
# Base 10, ASCII encoding of the byte length of the object
# contents, without leading zeros. The length value does not
# include the length of the <OBJECT_HEADER>.
: ( <DIGIT> )+
;
<SP>
# ASCII SPACE (0x20) character.
;
<NUL>
# ASCII NUL (0x00) character.
;
<LF>
# ASCII LF (0x0a) "line feed" character.
;
PACK FILE
---------
# The name of a pack file is "pack-${PACK_ID}.pack", where ${PACK_ID}
# is the hexidecimal encoding (lower case) of the SHA-1 digest of the
# sorted list of binary object IDs in the pack file without a separator
# between the object IDs. Initially, the ${PACK_ID} for a pack was not
# defined, making the value effectively random.
<PACK_FILE>
: <PACK_FILE_CONTENTS> <PACK_FILE_CHECKSUM>
;
<PACK_FILE_CONTENTS>
: "PACK" <PACK_VERSION> <PACK_OBJECT_COUNT>
( <PACKED_OBJECT_HEADER> <PACKED_OBJECT_DATA> )*
<PACK_FILE_CHECKSUM>
;
<PACK_VERSION>
# 32 bit, network byte order, binary integer indicating which
# version of the pack file format was used to create the pack
# file.
;
<PACK_OBJECT_COUNT>
# 32 bit, network byte order, binary integer containg the
# number of objects encoded in the pack file.
;
<PACK_FILE_CHECKSUM>
: _sha-1_digest_( <PACK_FILE_CONTENTS> )
;
<PACKED_OBJECT_HEADER>
# If the object type is not a DELTA_ENCODED object, the packed
# object data that follows is the deflated byte sequence of the
# object without the Git object header. The length value is the
# byte count of the inflated byte sequence of the object.
#
# If the object type is a DELTA_ENCODED object, what follows is
# the ID of the base object and the deflated delta data to
# transform the base object into the target object. The type of
# the target object is the same as that of the base object and
# the length value is the byte count of the inflated delta
# data. The base object may also be DELTA_ENCODED but cyclic
# base object chains are not permitted and the pack file MUST
# contain all base objects.
: <TYPE_AND_BASE128_SIZE>
;
<TYPE_AND_BASE128_SIZE>
# A compact, variable length, encoding of the packed object
# length and type. The first byte is comprised of 3 fields
# (where bit 0 is the least significant bit in a byte):
# bit 7: more flag
# bits 6-4: object type
# bits 3-0: least significant bits of the object
# length.
# If the more flag is set, the next byte contains more object
# length bits.
# The object types corresponding to the object type bits are:
# 6 5 4
# - - -
# 0 0 0 invalid: Reserved
# 0 0 1 COMMIT object
# 0 1 0 TREE object
# 0 1 1 BLOB object
# 1 0 0 TAG object
# 1 0 1 invalid: Reserved
# 1 1 0 invalid: Reserved
# 1 1 1 DELTA_ENCODED object
#
# If the more flag was set, the next byte will have more length
# bits and will be comprised to 2 fields:
# bit 7: more flag
# bits 6-0: 7 additional, more significant, bits of
# the object length
# If the more flag is set, the next byte contains more object
# length bits using the same encoding.
;
<PACKED_OBJECT_DATA>
: _deflate_( <DATA> )
| <BINARY_OBJ_ID> _deflate_( <DELTA_DATA> )
;
<DELTA_DATA>
# Size of the base object encoded as a base 128 number, least
# significant bits first, using bit 7 (the most significant
# bit) of each byte to indicate that more bits follow.
#
# Size of the result object encoded as a base 128 number, using
# the same method as used for the base object size.
#
# There will then be a sequence of delta hunks.
# Zero as the value of the first byte of a hunk in reserved.
#
# If bit 7 of the first byte of a delta hunk is not set, the
# hunk is an "insert" hunk and bits 0-6 specify the number of
# bytes to append to the output buffer from the hunk.
#
# If bit 7 of the first byte of a delta hunk is set, the hunk
# is a "copy" hunk and bits 0-6 specify how the remaining
# bytes in the hunk make up the base offset and length for the
# copy. The following C code demonstrate how to determine the
# base offset and length for the copy:
#
# /* - - - - - - - - - - - - *\
# | This reflects version 3 pack files |
# \* - - - - - - - - - - - - */
#
# byte *data = delta_hunk_start
# opcode = *data++
# off_t copy_offset= 0;
# size_t copy_length = 0;
#
# for (shift=i=0; i<4; i++) {
# if (opcode & 0x01) {
# copy_offset |= (*data++)<<shift;
# }
# opcode >>= 1;
# shift += 8;
# }
#
# for (shift=i=0; i<3; i++) {
# if (opcode & 0x01) {
# copy_length |= (*data++)<<shift;
# }
# opcode >>= 1;
# shift += 8;
# }
#
# if (!copy_length) {
# copy_length = 1<<16;
# }
#
# For version 2 pack files, the size of a copy is limited to
# 64K bytes or less and bit 6 of the opcode byte is set if the
# source of the copy is from the buffer of the result object
# instead of the the base object.
#
# It's unknown if any version 2 pack files were created with
# bit 6 set in the opcode byte; however, the change that added
# support for version 3 pack files removed the code that would
# change the copy source to the result buffer.
#
# /* - - - - - - - - - - - - *\
# | This reflects version 2 pack files |
# \* - - - - - - - - - - - - */
#
# byte *data = delta_hunk_start
# opcode = *data++
# off_t copy_offset= 0;
# size_t copy_length = 0;
#
# for (shift=i=0; i<4; i++) {
# if (opcode & 0x01) {
# copy_offset |= (*data++)<<shift;
# }
# opcode >>= 1;
# shift += 8;
# }
#
# for (shift=i=0; i<2; i++) {
# if (opcode & 0x01) {
# copy_length |= (*data++)<<shift;
# }
# opcode >>= 1;
# shift += 8;
# }
#
# if (!copy_length) {
# copy_length = 1<<16;
# }
#
# copy_from_result = opcode & 0x01
#
;
PACK INDEX
----------
# The name of a pack file index is "pack-${PACK_ID}.idx", where
# ${PACK_ID} is the hexidecimal encoding (lower case) of the SHA-1
# digest of the sorted list of binary object IDs in the pack file
# without a separator between the object IDs. Initially, the ${PACK_ID}
# for a pack was not defined, making the value effectively random.
<PACK_INDEX>
: <PACK_INDEX_CONTENTS> <PACK_INDEX_CHECKSUM>
;
<PACK_INDEX_CONTENTS>
: ( <INDEX_PARTIAL_COUNT> ){256}
( <PACK_OBJECT_OFFSET> <BINARY_OBJ_ID> )*
<PACK_FILE_CHECKSUM>
;
<INDEX_PARTIAL_COUNT>
# 32 bit, network byte order, binary integer of the count of
# objects in the pack file with the first byte of the object
# ID less than or equal to the index of the count, starting
# from zero.
;
<PACK_OBJECT_OFFSET>
# 32 bit, network byte order, binary integer giving the offset,
# in bytes from the begining of the pack file, where the
# encoding of the object starts.
;
<PACK_INDEX_CHECKSUM>
: _sha-1_digest_( <PACK_INDEX_CONTENTS> )
;
INDEX FILE (CACHE)
------------------
<INDEX_FILE>
: <INDEX_FILE_FORMAT_V1>
| <INDEX_FILE_FORMAT_V2>
;
<INDEX_FILE_FORMAT_V1>
# This format is no longer supported.
: <INDEX_HEADER> <INDEX_CHECKSUM> <INDEX_CONTENTS>
;
<INDEX_FILE_FORMAT_V2>
: <INDEX_HEADER> <EXTENDED_INDEX_CONTENTS> <EXTENED_CHECKSUM>
;
<INDEX_HEADER>
: "DIRC" <INDEX_FILE_VERSION> <INDEX_ENTRY_COUNT>
;
<INDEX_FILE_VERSION>
# 32 bit, network byte order, binary integer indicating which
# version of the index file format was used to create the
# index file.
;
<INDEX_ENTRY_COUNT>
# 32 bit, network byte order, binary integer containg the
# number of index entries in the index file.
;
<EXTENDED_CHECKSUM>
: _sha-1_digest_( <EXTENDED_INDEX_CONTENTS> )
;
<INDEX_CHECKSUM>
: _sha-1_digest_( <INDEX_CONTENTS> )
;
<INDEX_CONTENTS>
: ( <INDEX_ENTRY> )*
;
<EXTENDED_INDEX_CONTENTS>
: <INDEX_CONTENTS> <INDEX_CONTENTS_EXTENSIONS>
;
<INDEX_ENTRY>
: <INDEX_ENTRY_STAT_INFO>
<ENTRY_ID>
<ENTRY_FLAGS>
<ENTRY_NAME>
;
<INDEX_ENTRY_STAT_INFO>
# These fields are used as a part of a heuristic to determine
# if the file system entity associated with this entry has
# changed. The names are very *nix centric but the exact
# contents of each field have no meaning to Git, besides exact
# match, except for the <ENTRY_MODE> and <ENTRY_SIZE> fields.
: <ENTRY_CTIME>
<ENTRY_MTIME>
<ENTRY_DEV>
<ENTRY_INODE>
<ENTRY_MODE>
<ENTRY_UID>
<ENTRY_GID>
<ENTRY_SIZE>
;
<ENTRY_CTIME>
# The timestamp of the last status change of the associated
# file system entity.
: <ENTRY_TIME>
;
<ENTRY_MTIME>
# The timestamp of the last modification of the associated
# file system entity.
: <ENTRY_TIME>
;
<ENTRY_TIME>
: <TIME_LSB32> <TIME_NSEC>
;
<TIME_LSB32>
# 32 bit, network byte order, binary integer containg the lower
# 32 bits of the entry (file or symbolic link) timestamp.
;
<TIME_NSEC>
# 32 bit, network byte order, binary integer containg the lower
# 32 bits of the entry (file or symbolic link) more precise
# timestamp, if available.
;
<ENTRY_DEV>
# 32 bit, network byte order, binary integer containg the lower
# 32 bits of the entry (file or symbolic link) file system
# device identifier. Use of this field is a compile time
# option.
;
<ENTRY_INODE>
# 32 bit, network byte order, binary integer containg the lower
# 32 bits of the entry (file or symbolic link) inode number, or
# equivalent.
;
<ENTRY_MODE>
# 32 bit, network byte order, binary integer containg the lower
# 32 bits of the entry (file or symbolic link) file system
# entity type and permissions.
;
<ENTRY_UID>
# 32 bit, network byte order, binary integer containg the lower
# 32 bits of the entry (file or symbolic link) file system
# entity owner identifier.
;
<ENTRY_GID>
# 32 bit, network byte order, binary integer containg the lower
# 32 bits of the entry (file or symbolic link) file system
# entity group identifier, or equivalent.
;
<ENTRY_SIZE>
# 32 bit, network byte order, binary integer containg the lower
# 32 bits of the entry (file or symbolic link) size.
;
<ENTRY_ID>
# Object ID of the of the file system entity contents.
;
<ENTRY_FLAGS>
# 16 bit, network byte order, binary integer.
# bits 15-14 Reserved
# bits 13-12 Entry stage
# bits 11-0 Name byte length
#
# See git-read-tree(1) for a description of how the stage
# field is used.
;
<ENTRY_NAME>
# File system entity name. Path is normalized and relative to
# the working directory.
;
<INDEX_CONTENTS_EXTENSIONS>
: ( <INDEX_EXTENSION> )*
;
<INDEX_EXTENSION>
: <INDEX_EXTENSION_HEADER>
<INDEX_EXTENSION_DATA>
;
<INDEX_EXTENSION_HEADER>
: <INDEX_EXTENSION_NAME> <INDEX_EXTENSION_DATA_SIZE>
;
<INDEX_EXTENSION_NAME>
# 4 byte sequence identifying how the <INDEX_EXTENSION_DATA>
# should be interpreted. The first byte having a value greater
# than or equal to the ASCII character 'A' (0x41) and less than
# or equal to the ASCII character 'Z' (0x5a).
;
<INDEX_EXTENSION_DATA_SIZE>
# 32 bit, network byte order, binary integer containg the
# length of the <INDEX_EXTENSION_DATA> byte sequence.
;
<INDEX_EXTENSION_DATA>
# Sequence of bytes.
;
^ permalink raw reply
* Re: Creating objects manually and repack
From: Jon Smirl @ 2006-08-05 5:40 UTC (permalink / raw)
To: Shawn Pearce; +Cc: Martin Langhoff, Linus Torvalds, git
In-Reply-To: <20060805052135.GA18679@spearce.org>
On 8/5/06, Shawn Pearce <spearce@spearce.org> wrote:
> I'm almost done with what I'm calling `git-fast-import`. It takes
> a stream of blobs on STDIN and writes the pack to a file, printing
> SHA1s in hex format to STDOUT. The basic format for STDIN is a 4
> byte length (native format) followed by that many bytes of blob data.
> It prints the SHA1 for that blob to STDOUT, then waits for another
> length.
>
> It naively deltas each object against the prior object, thus it
> would be best to feed it one ,v file at a time working from the most
> recent revision back to the oldest revision. This works well for
> an RCS file as that's the natural order to process the file in. :-)
I am already doing this.
> When done you close STDIN and it'll rip through and update the pack
> object count and the trailing checksum. This should let you pack
> the entire repository in delta format using only two passes over the
> data: one to write out the pack file and one to compute its checksum.
Thinking about this some more, the existing repack code could be made
to work with minor changes. I would like to feed repack 1M revisions
which are sorted by file and then newest to oldest. The problem is
that my expanded revs take up 12GB disk space.
How about adding a flag to repack that simply says delete the objects
when done with them? I'd still create all of the objects on disk.
Repack would assume that they have at least been sorted by filename.
So repack could read in object names until it sees a change in the
file name, sort them by size, deltafy, write out the pack and then
delete the objects from that batch. Then repeat this process for the
next file name on stdin.
I'm making two assumptions, first that blocks from a deleted file
don't get written to disk. And that by deleting the file the file
system will use the same blocks over and over. If those assumptions
are close to being true then the cache shouldn't thrash. They don't
have to be totally true, close is good enough.
Of course eliminating the files all together will be even faster.
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: Creating objects manually and repack
From: Shawn Pearce @ 2006-08-05 5:46 UTC (permalink / raw)
To: Jon Smirl; +Cc: Martin Langhoff, Linus Torvalds, git
In-Reply-To: <20060805052135.GA18679@spearce.org>
[-- Attachment #1: Type: text/plain, Size: 2432 bytes --]
Shawn Pearce <spearce@spearce.org> wrote:
> Jon Smirl <jonsmirl@gmail.com> wrote:
> > On 8/5/06, Martin Langhoff <martin.langhoff@gmail.com> wrote:
> > >On 8/5/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> > >> On 8/4/06, Linus Torvalds <torvalds@osdl.org> wrote:
> > >> > and you're basically all done. The above would turn each *,v file into
> > >a
> > >> > *-<sha>.pack/*-<sha>.idx file pair, so you'd have exactly as many
> > >> > pack-files as you have *,v files.
> > >>
> > >> I'll end up with 110,000 pack files.
> > >
> > >Then just do it every 100 files, and you'll only have 1,100 pack
> > >files, and it'll be fine.
> >
> > This is something that has to be tuned. If you wait too long
> > everything spills out of RAM and you go totally IO bound for days. If
> > you do it too often you end up with too many packs and it takes a day
> > to repack them.
> >
> > If I had a way to pipe the all of the objects into repack one at a
> > time without repack doing multiple passes none of this tuning would be
> > necessary. In this model the standalone objects never get created in
> > the first place. The fastest IO is IO that has been eliminated.
>
> I'm almost done with what I'm calling `git-fast-import`.
OK, now I'm done. I'm attaching the code. Toss it into the Makefile
as git-fast-import and recompile.
I tested it with the following Perl script, feeding the Perl script
a list of files that I wanted blobs for on STDIN:
while (<>) {
chop;
print pack('L', -s $_);
open(F, $_);
my $buf;
print $buf while read(F,$buf,128*1024) > 0;
close F;
}
This gave me an execution order of:
find . -name '*.c' | perl test.pl | git-fast-import in.pack
git-index-pack in.pack
at which point in.pack claims to be a completely valid pack with an
index of in.idx. Move these into .git/objects/pack, generate trees
and commits, and run git-repack -a -d. If the order you feed the
objects to git-fast-import in is reasonable (do one RCS file at a
time, feed most recent to least recent revisions) you may not get
any major benefit from using -f during your final repack.
The code for git-fast-import could probably be tweaked to accept
trees and commits too, which would permit you to stream the entire
CVS repository into a single pack file. :-)
I can't help you decompress the RCS files faster, but hopefully
this will help you generate the GIT pack faster. Hopefully you
can make use of it!
--
Shawn.
[-- Attachment #2: fast-import.c --]
[-- Type: text/x-csrc, Size: 4659 bytes --]
#include "builtin.h"
#include "cache.h"
#include "object.h"
#include "blob.h"
#include "delta.h"
#include "pack.h"
#include "csum-file.h"
static int max_depth = 10;
static unsigned long object_count;
static int packfd;
static int current_depth;
static void *lastdat;
static unsigned long lastdatlen;
static unsigned char lastsha1[20];
static ssize_t yread(int fd, void *buffer, size_t length)
{
ssize_t ret = 0;
while (ret < length) {
ssize_t size = xread(fd, (char *) buffer + ret, length - ret);
if (size < 0) {
return size;
}
if (size == 0) {
return ret;
}
ret += size;
}
return ret;
}
static ssize_t ywrite(int fd, void *buffer, size_t length)
{
ssize_t ret = 0;
while (ret < length) {
ssize_t size = xwrite(fd, (char *) buffer + ret, length - ret);
if (size < 0) {
return size;
}
if (size == 0) {
return ret;
}
ret += size;
}
return ret;
}
static unsigned long encode_header(enum object_type type, unsigned long size, unsigned char *hdr)
{
int n = 1;
unsigned char c;
if (type < OBJ_COMMIT || type > OBJ_DELTA)
die("bad type %d", type);
c = (type << 4) | (size & 15);
size >>= 4;
while (size) {
*hdr++ = c | 0x80;
c = size & 0x7f;
size >>= 7;
n++;
}
*hdr = c;
return n;
}
static void write_blob (void *dat, unsigned long datlen)
{
z_stream s;
void *out, *delta;
unsigned char hdr[64];
unsigned long hdrlen, deltalen;
if (lastdat && current_depth < max_depth) {
delta = diff_delta(lastdat, lastdatlen,
dat, datlen,
&deltalen, 0);
} else
delta = 0;
memset(&s, 0, sizeof(s));
deflateInit(&s, zlib_compression_level);
if (delta) {
current_depth++;
s.next_in = delta;
s.avail_in = deltalen;
hdrlen = encode_header(OBJ_DELTA, deltalen, hdr);
if (ywrite(packfd, hdr, hdrlen) != hdrlen)
die("Can't write object header: %s", strerror(errno));
if (ywrite(packfd, lastsha1, sizeof(lastsha1)) != sizeof(lastsha1))
die("Can't write object base: %s", strerror(errno));
} else {
current_depth = 0;
s.next_in = dat;
s.avail_in = datlen;
hdrlen = encode_header(OBJ_BLOB, datlen, hdr);
if (ywrite(packfd, hdr, hdrlen) != hdrlen)
die("Can't write object header: %s", strerror(errno));
}
s.avail_out = deflateBound(&s, s.avail_in);
s.next_out = out = xmalloc(s.avail_out);
while (deflate(&s, Z_FINISH) == Z_OK)
/* nothing */;
deflateEnd(&s);
if (ywrite(packfd, out, s.total_out) != s.total_out)
die("Failed writing compressed data %s", strerror(errno));
free(out);
if (delta)
free(delta);
}
static void init_pack_header ()
{
const char* magic = "PACK";
unsigned long version = 2;
unsigned long zero = 0;
version = htonl(version);
if (ywrite(packfd, (char*)magic, 4) != 4)
die("Can't write pack magic: %s", strerror(errno));
if (ywrite(packfd, &version, 4) != 4)
die("Can't write pack version: %s", strerror(errno));
if (ywrite(packfd, &zero, 4) != 4)
die("Can't write 0 object count: %s", strerror(errno));
}
static void fixup_header_footer ()
{
SHA_CTX c;
char hdr[8];
unsigned char sha1[20];
unsigned long cnt;
char *buf;
size_t n;
if (lseek(packfd, 0, SEEK_SET) != 0)
die("Failed seeking to start: %s", strerror(errno));
SHA1_Init(&c);
if (yread(packfd, hdr, 8) != 8)
die("Failed reading header: %s", strerror(errno));
SHA1_Update(&c, hdr, 8);
fprintf(stderr, "%lu objects\n", object_count);
cnt = htonl(object_count);
SHA1_Update(&c, &cnt, 4);
if (ywrite(packfd, &cnt, 4) != 4)
die("Failed writing object count: %s", strerror(errno));
buf = xmalloc(128 * 1024);
for (;;) {
n = xread(packfd, buf, 128 * 1024);
if (n <= 0)
break;
SHA1_Update(&c, buf, n);
}
free(buf);
SHA1_Final(sha1, &c);
if (ywrite(packfd, sha1, sizeof(sha1)) != sizeof(sha1))
die("Failed writing pack checksum: %s", strerror(errno));
}
int main (int argc, const char **argv)
{
packfd = open(argv[1], O_RDWR|O_CREAT|O_TRUNC, 0666);
if (packfd < 0)
die("Can't create pack file %s: %s", argv[1], strerror(errno));
init_pack_header();
for (;;) {
unsigned long datlen;
int hdrlen;
void *dat;
char hdr[128];
unsigned char sha1[20];
SHA_CTX c;
if (yread(0, &datlen, 4) != 4)
break;
dat = xmalloc(datlen);
if (yread(0, dat, datlen) != datlen)
break;
hdrlen = sprintf(hdr, "blob %lu", datlen) + 1;
SHA1_Init(&c);
SHA1_Update(&c, hdr, hdrlen);
SHA1_Update(&c, dat, datlen);
SHA1_Final(sha1, &c);
write_blob(dat, datlen);
object_count++;
printf("%s\n", sha1_to_hex(sha1));
fflush(stdout);
if (lastdat)
free(lastdat);
lastdat = dat;
lastdatlen = datlen;
memcpy(lastsha1, sha1, sizeof(sha1));
}
fixup_header_footer();
close(packfd);
return 0;
}
^ permalink raw reply
* Re: Git files data formats documentation
From: Jon Smirl @ 2006-08-05 5:48 UTC (permalink / raw)
To: gitzilla; +Cc: git
In-Reply-To: <44D42F0D.3040707@gmail.com>
You might make some notes about old format headers and new format ones
and the use_legacy_headers flag.
I started off looking at packs so I knew about TYPE_AND_BASE128_SIZE.
Next I wanted to write objects so I looked at sha1_file.c. If you
don't look at the code closely write_binary_header() will lead you to
believe that object files use TYPE_AND_BASE128_SIZE. It took me a
couple of hours to notice use_legacy_headers and discover that it
defaults to on.
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: Creating objects manually and repack
From: Shawn Pearce @ 2006-08-05 5:52 UTC (permalink / raw)
To: Jon Smirl; +Cc: Martin Langhoff, Linus Torvalds, git
In-Reply-To: <9e4733910608042240u581dd23q3859ebcfe4268ce2@mail.gmail.com>
Jon Smirl <jonsmirl@gmail.com> wrote:
> How about adding a flag to repack that simply says delete the objects
> when done with them? I'd still create all of the objects on disk.
> Repack would assume that they have at least been sorted by filename.
> So repack could read in object names until it sees a change in the
> file name, sort them by size, deltafy, write out the pack and then
> delete the objects from that batch. Then repeat this process for the
> next file name on stdin.
>
> I'm making two assumptions, first that blocks from a deleted file
> don't get written to disk. And that by deleting the file the file
> system will use the same blocks over and over. If those assumptions
> are close to being true then the cache shouldn't thrash. They don't
> have to be totally true, close is good enough.
>
> Of course eliminating the files all together will be even faster.
See the email I just sent you. The only file being written is the
pack file that's being generated. No temporary files, no temporary
inodes, no temporary blocks. Only two passes over the data: one to
write it out and a second to generate the SHA1. I do two passes
vs. keep it all in memory to prevent the program from blowing out
on extremely large inputs.
It may be possible to tweak git-pack-objects to get what you propose
above, but to be honest I think the git-fast-import I just sent
was easier, especially as it avoids the temporary loose object stage.
--
Shawn.
^ permalink raw reply
* Re: Git files data formats documentation
From: Junio C Hamano @ 2006-08-05 6:51 UTC (permalink / raw)
To: A Large Angry SCM; +Cc: git
In-Reply-To: <44D42F0D.3040707@gmail.com>
Good documentation, but some nitpicks are needed before it hits
Documentation/technical/ part of the source tree.
> <TREE_ENTRIES>
> # Tree entries are sorted by the byte sequence that comprises
> # the entry name.
> : ( <TREE_ENTRY> )*
> ;
Not quite. An entry for a subtree is sorted as if a '/' is
suffixed to its name.
$ git ls-tree $T
100644 blob 2398e9f8892812607f5eee6ed0d5712c2e3de197 a-
100644 blob 7f07527a80bd8c2b1c5087d7ccfe61073b068374 a-b
040000 tree 23fddf6a57ff3ba98aa93fb71431276c3f1a3c40 a
100644 blob 2afe6dcc5466068b8dcc7263cece05d2adf044fe a=
100644 blob efc73add7dd868242a66faf2a59b145f2a60b834 a=b
This is, by the way, consistent with the order of cache entries
in the index file.
$ git ls-files -s
100644 2398e9f8892812607f5eee6ed0d5712c2e3de197 0 a-
100644 7f07527a80bd8c2b1c5087d7ccfe61073b068374 0 a-b
100644 0ee729686ab2a0074639c5f64930648571e7c4b2 0 a/b
100644 2afe6dcc5466068b8dcc7263cece05d2adf044fe 0 a=
100644 efc73add7dd868242a66faf2a59b145f2a60b834 0 a=b
> <TREE_ENTRY>
> # The type of the object referenced MUST be appropriate for
> # the mode. Regular files and symbolic links reference a BLOB
> # and directories reference a TREE.
> : <OCTAL_MODE> <SP> <NAME> <NUL> <BINARY_OBJ_ID>
> ;
As you correctly explain later, OCTAL_MODE must be minimal; "git
ls-tree" output says 040000 in the above example, but the actual
object records it as 40000.
> <TAG_CONTENTS>
> : "object" <SP> <HEX_OBJ_ID> <LF>
> "type" <SP> <NONTAG_OBJ_TYPE> <LF>
> "tag" <SP> <TAG_NAME> <LF>
> <LF>
> <DATA>
> ;
A tag can tag another tag (think of chain of trust), so what
follows "type" does not have to be NONTAG_OBJ_TYPE.
> <OCTAL_MODE>
> # Octal encoding, without prefix, of the file system object
> # type and permission bits. The bit layout is according to the
> # POSIX standard, with only regular files, directories, and
> # symbolic links permitted. The actual permission bits are
> # all zero except for regular files. The only permission bit
> # of any consequence to Git is the owner executable bit. By
> # default, the permission bits for files will be either 0644
> # or 0755, depending on the owner executable bit.
> ;
It's not really "by default" -- more like "by definition", since
there is no way for the program to use something different. We
used to record non-canonical modes in ancient versions of git,
but I think fsck-objects would warn on objects created that way.
> <NONTAG_OBJ_TYPE>
> : "BLOB"
> | "TREE"
> | "COMMIT"
> ;
Drop this definition, and make the literals part of <OBJ_TYPE>,
after lowercasing them ;-).
> <OBJ_TYPE>
> : <NONTAG_OBJ_TYPE>
> | "TAG"
> ;
> PACK FILE
> ---------
> # The name of a pack file is "pack-${PACK_ID}.pack", where ${PACK_ID}
> # is the hexidecimal encoding (lower case) of the SHA-1 digest of the
> # sorted list of binary object IDs in the pack file without a separator
> # between the object IDs. Initially, the ${PACK_ID} for a pack was not
> # defined, making the value effectively random.
Although the really core level does not care, a PACK_ID is
required to be unique (within a object store and its alternates)
40-byte hexadecimal for http commit walker to work properly.
BTW, I still have a patch to tighten the check to enforce this
as part of the consistency check.
> <PACKED_OBJECT_DATA>
> : _deflate_( <DATA> )
> | <BINARY_OBJ_ID> _deflate_( <DELTA_DATA> )
> ;
It might be cleaner to separate this definition into two. That
is, one packed object is either non-delta-type base128 type-length
followed by deflated data, or delta-type base128 type-length
followed by base object id followed by deflated delta.
> PACK INDEX
> ----------
> # The name of a pack file index is "pack-${PACK_ID}.idx", where
> # ${PACK_ID} is the hexidecimal encoding (lower case) of the SHA-1
> # digest of the sorted list of binary object IDs in the pack file
> # without a separator between the object IDs. Initially, the ${PACK_ID}
> # for a pack was not defined, making the value effectively random.
I would not repeat ", where ${PACK_ID} is..." part, which was
done in the description of the pack file. Rather, ", where
${PACK_ID} is same as the .pack file the .idx file corresponds
to", would be more appropriate.
> <INDEX_PARTIAL_COUNT>
> # 32 bit, network byte order, binary integer of the count of
> # objects in the pack file with the first byte of the object
> # ID less than or equal to the index of the count, starting
> # from zero.
> ;
Linus and I call this part "fan-out".
> <ENTRY_NAME>
> # File system entity name. Path is normalized and relative to
> # the working directory.
> ;
Did you mention that the index entries are sorted by name?
> <INDEX_EXTENSION_NAME>
> # 4 byte sequence identifying how the <INDEX_EXTENSION_DATA>
> # should be interpreted. The first byte having a value greater
> # than or equal to the ASCII character 'A' (0x41) and less than
> # or equal to the ASCII character 'Z' (0x5a).
> ;
This is not true, but the code needs better comments. The
intention is that an extended section whose name starts with a
capital letter (such as "cache-tree extension" whose name is
"TREE") is purely optional, and if a software of different
version does not understand it, it can still safely keep using
the rest of the index. If somebody introduces a new extended
section that _must_ be interpreted in order to fully understand
what the index file records, such an extended section can signal
that by having a name that do not start with a capital. A
version of the software that does understand such extended
sections would have a case arm that covers such a name in the
switch statement you took this 'A' .. 'Z' from.
^ permalink raw reply
* Re: [PATCH 4/4] gitweb: No periods for error messages
From: Junio C Hamano @ 2006-08-05 7:03 UTC (permalink / raw)
To: Luben Tuikov; +Cc: git, Jakub Narebski
In-Reply-To: <20060805010307.17651.qmail@web31807.mail.mud.yahoo.com>
Luben Tuikov <ltuikov@yahoo.com> writes:
> --- Junio C Hamano <junkio@cox.net> wrote:
>
>> I tend to prefer ending each sentence with a full stop.
>
> I've never seen this in, among other things,
> - kernel messages
> - errno messages
> - web server messages
> - RFC text describing error messages, (web services),
> - etc.
Comparing
git grep \( -e die --or -e die \) --and -e '\."' -- '*.c'
and
git grep \( -e die --or -e die \) --and --not -e '\."' -- '*.c'
tells me that we omit periods, mostly.
> I was going for was consistency. I'd say apply Jacob's [6/5].
Well, [6/5] is on top of [1/5] which had the problem of
parroting unsanitized user input, so I'd rather use a clean
patch that does only this error message cleanups.
^ permalink raw reply
* Re: [PATCH] git-status: colorize status output
From: Greg KH @ 2006-08-05 8:16 UTC (permalink / raw)
To: Jeff King; +Cc: git
In-Reply-To: <20060805032135.GA11244@coredump.intra.peff.net>
On Fri, Aug 04, 2006 at 11:21:35PM -0400, Jeff King wrote:
> On Fri, Aug 04, 2006 at 11:14:19PM -0400, Jeff King wrote:
>
> > The idea is that red things indicate a potential mistake on the part of the
> > user (e.g., forgetting to update a file, forgetting to git-add a file).
>
> I actually wanted to do this because I started syntax-highlighting the
> git-commit message in vim. I found myself catching simple mistakes in
> commits, like the ones I mentioned above, before committing, saving me
> from doing an --amend. Then I got so hooked on it I wanted the
> colorization everytime I ran git-status.
>
> If anyone is interested in the vim syntax highlighting, it is below.
> Copy the file to $HOME/.vim/syntax/gitcommit.vim and add the following
> line to your .vimrc:
> autocmd BufNewFile,BufRead COMMIT_EDITMSG set filetype=gitcommit
Ah, very nice, thanks for this, makes commits much easier to read now.
greg k-h
^ permalink raw reply
* Enjoy the newest It will be great
From: Chi @ 2006-08-05 9:29 UTC (permalink / raw)
To: git
Good day Bro,
Rock hard manhood, multiple explosions and several times more semen volume
Get a several month supply of everything you need – in seconds
World famous brands which keep men happy all over the world
Enter: http://www.remeltib.com
The prices are really low and the quality it truly very high!
^ permalink raw reply
* Re: [PATCH] git-status: colorize status output
From: Junio C Hamano @ 2006-08-05 9:28 UTC (permalink / raw)
To: Jeff King; +Cc: git
In-Reply-To: <20060805031418.GA11102@coredump.intra.peff.net>
Jeff King <peff@peff.net> writes:
> + M ) color $1; echo "modified: $name"; uncolor $1;;
> + D*) color $1; echo "deleted: $name"; uncolor $1;;
> + T ) color $1; echo "1change: $name"; uncolor $1;;
Is "1" a typo?
> + C*) color $1; echo "copied: $name -> $newname"; uncolor $1;;
> + R*) color $1; echo "renamed: $name -> $newname"; uncolor $1;;
> + A*) color $1; echo "new file: $name"; uncolor $1;;
> + U ) color $1; echo "unmerged: $name"; uncolor $1;;
> + O ) color $1; echo "$name"; uncolor $1;;
^ permalink raw reply
* Re: [RFC][PATCH] Branch history
From: Eric W. Biederman @ 2006-08-05 9:30 UTC (permalink / raw)
To: Shawn Pearce; +Cc: git
In-Reply-To: <20060805031821.GB18223@spearce.org>
Shawn Pearce <spearce@spearce.org> writes:
> "Eric W. Biederman" <ebiederm@xmission.com> wrote:
>>
>> The problem:
>> git-rebase, stgit and the like destructively edit the commit history
>> on a branch. Making it a challenge to go back to a known good point.
>>
>> revlog and the like sort of help this but they don't address the
>> issues that they capture irrelevant points and are not git-prune safe.
>
> How are the points irrelevant? Each commit/rebase/am/update-ref
> is recorded. That's each change to the branch head. It appears
> as though you are mainly interested in tracking across rebases,
> which a reflog would do, assuming you filtered the events down to
> only those caused by rebase and ignored the others.
It tracks each change, it does not track the changes that humans find
interesting. That can easily be a lot of noise.
I don't want to see every time a head is updated any more than I want
single keystroke level version control. Way too much uninteresting
detail.
> But yea, a reflog is not prune-safe, but it wouldn't be hard to
> modify git-prune to also consider the reflog associated with
> a ref if its using that ref as a root that must be preserved.
> Assuming anyone really wants that as a feature...
I do. I also want history I can clone between repositories.
I have times I have had to look 9 months back to see where I accidentally
dropped a patch.
>> After thinking about the problem some more I believe I have found
>> a rather simple solution to the problem of keeping branch history.
>>
>> For each branch you want to keep the history of keep 2 branches.
>> A normal working branch, and a second archive branch that records
>> the history of the branch you are editing.
>
> It would appear as though you are really only tracking rebase events,
> as everything else done on the branch is preserved since the work
> branch is itself parent #1 for the archive branch commit. So the
> archive branch shows every commit ever done along the main branch,
> but also shows itself joining back quite frequently. Further if you
> archive away the work branch without during a rebase since the last
> archive then there's really nothing happening except saving a tag
> (but as a commit!) on the archive branch.
True. But that is largely the wrong way to think about it. I am
saving away a branch at times it is interesting to a human being.
There are also other tools and other methods of editing a branch
besides git-rebase.
> This creates for a rather messy history, and is more-or-less what
> pg does when patches get pushed onto a stack and they can't be
> pushed by a simple fast-forward operation. Reading this history
> in gitk is "interesting" at best. This is the main reason I've
> been trying to write `tb` (a topic branch manager, fashioned after
> Junio's TO script) but I can't seem to find enough time to get it
> finished.
I just took a quick look at pg, and while the mechanism may be
similar I believe the goals are fundamentally different. I am
trying to record the history at points human beings care about,
pg seems to do something automatically behind the scenes, with
the existing model.
The points I am recording the history are points at which I want a
human commit message, because these are points in time meaningful to
me. The ideal companion would be something that could just walk my
branch history and pull it out. So when generating an overview
message I could easily generate a summary of how I had been editing my
patches.
>> The neat thing is that it gives an immutable history of a branch that
>> is actively being edited. So if you export your archive branch people
>> will never see time roll backward.
>
> Right. That's an interesting way of handling it, but that branch
> is also quite messy as its full of merge commits. Although it may
> be useful to export its going to carry along with it all of the bad
> edits and prior rebases made on that branch. You probably wouldn't
> want to merge that branch into a mainline, which means that branch
> is likely to be discarded at some point in the future. When that
> happens then nobody can track it anymore and that immutable history
> just got mutated out of existance.
Yes. But it is interesting until it gets merged into mainline, and
keeping around in the developers own archives. Mistakes can be
interesting. I don't expect that there will be a need for keeping
the mistakes after a branch is perfected and merged into mainline.
Until the branch is perfected though I fully expect there to be bad
branch history edits that need to be fixed.
The point at which the immutable history goes out of existence is
the point where the branch stops being interesting as an entity
in it's own right. So I think that is exactly the right behavior.
> I think the right way to deal with these types of branches is to
> publicly publish whether or not the branch is going to be expected
> to roll backwards in time (due to a rebase type of event) then
> let clients always update those branches during pulls, rather
> than needing to explicitly mark them with '+' on the client side.
Not if part of the problem is distributing the work of coming up
with a perfect patch set. If you don't distribute the history
it is hard to see what someone has really changed. You can't help
me undo a branch editing mistake if you don't have the previous
version of the branch. It is hard to verify I actually fixed what
you are concerned about if you don't have the old version to compare
against.
> Further good remege tools (git-rerere on steriods) would help
> re-resolve conflicts resulting from continous rebasing. This would
> make it easier to maintain such a branch and carry the thing forward;
> or to leave it on its original base but to continously remerge
> it and the current mainline into a temporary working branch for
> testing purposes.
Rebase is not the primary operation. I have one basic branch that
I have 10 copies of against v2.6.18-rc3. Refactoring, debugging,
and perfecting patches is a much more interesting event than rebasing.
Although rebasing does happen as well.
If you look at the -mm tree it tends to have 2-3 releases before
getting rebased.
> This is largely the policy that Junio uses for the `pu` and
> the `next` branches, as well as for the topic branches that he
> carries for everyone else doing GIT development. It appears to be
> working rather well, but it certainly could be streamlined better.
> My git-rerere2 and tb tools are an attempt to do this, but sadly
> they aren't in a useful state yet. Maybe because they are both
> far more complex then what you are doing here. :-)
To some extent I have a very interesting subset of kernel development.
Most of my changes are to systems that I am not a maintainer of.
Most of my changes are substantial, and scary because they touch
fundamental things. Most of my change involve many interdependent
patches, so topic branches cannot solve my problems.
For edits stgit git-rebase certainly can help, and I clearly
anticipate better tools in that vein, as well as better tools
for dealing with topic branches.
But that isn't the problem I am trying to solve here. I am trying
to implement version control for branch edits, (with maximum
capability with the existing git).
> Nonethless it is an interesting contribution. Thank you for taking
> the time to send it.
Welcome.
I think by making branch edit history something fundamental, we
achieve some fairly substantial things.
- We don't care about how the operations to edit a branch are
implemented, making them simpler to write.
- We begin to allow distributed branch editing.
- Branches become primary objects we can work with.
Hopefully I have stirred up the pot enough to allow some interesting
things.
Eric
^ permalink raw reply
* Re: [PATCH] git-status: colorize status output
From: Jeff King @ 2006-08-05 10:21 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vpsffedkw.fsf@assigned-by-dhcp.cox.net>
On Sat, Aug 05, 2006 at 02:28:31AM -0700, Junio C Hamano wrote:
> > + M ) color $1; echo "modified: $name"; uncolor $1;;
> > + D*) color $1; echo "deleted: $name"; uncolor $1;;
> > + T ) color $1; echo "1change: $name"; uncolor $1;;
> Is "1" a typo?
Oops, yes, not sure how that got in there.
-Peff
^ permalink raw reply
* Re: [PATCH] git-status: colorize status output
From: Jeff King @ 2006-08-05 10:44 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <20060805102117.GA28348@sigio.intra.peff.net>
On Sat, Aug 05, 2006 at 06:21:18AM -0400, Jeff King wrote:
> > > + T ) color $1; echo "1change: $name"; uncolor $1;;
> > Is "1" a typo?
> Oops, yes, not sure how that got in there.
Oh, I see. It should be 'typechange' in case you didn't reference
against the original version.
-Peff
^ permalink raw reply
* [PATCH 0/9] gitweb: First patch corrected and split into separate patches
From: Jakub Narebski @ 2006-08-05 10:51 UTC (permalink / raw)
To: Jakub Narebski, git
In-Reply-To: <eb0oiu$sj1$1@sea.gmane.org>
Jakub Narebski wrote:
> I'm sorry for unrelated changes (the commit could be probably split
> into four).
This series splits unrelated changes into separate patches, does not add
controversial adding (unescaped) value of invalid/errorneous parameter to
error message, and correct errors noticed during creation of this series.
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* [PATCH 1/9] gitweb: Separate input validation and dispatch, add comment about opml action
From: Jakub Narebski @ 2006-08-05 10:55 UTC (permalink / raw)
To: git
In-Reply-To: <44d47813.36251c31.2553.3cf7@mx.gmail.com>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
gitweb/gitweb.perl | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 58eb5b1..8b53fe6 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -71,6 +71,7 @@ if (! -d $git_temp) {
mkdir($git_temp, 0700) || die_error("Couldn't mkdir $git_temp");
}
+# ======================================================================
# input validation and dispatch
our $action = $cgi->param('a');
if (defined $action) {
@@ -78,6 +79,7 @@ if (defined $action) {
undef $action;
die_error(undef, "Invalid action parameter.");
}
+ # action which does not check rest of parameters
if ($action eq "opml") {
git_opml();
exit;
--
1.4.1.1
^ permalink raw reply related
* Re: [PATCH] git-status: colorize status output
From: Matthias Lederhofer @ 2006-08-05 10:59 UTC (permalink / raw)
To: Jeff King; +Cc: git
In-Reply-To: <20060805031418.GA11102@coredump.intra.peff.net>
Jeff King <peff@peff.net> wrote:
> The git-status output can sometimes be very verbose, making it difficult to
> quickly see whether your files are updated in the index. This adds 4 levels
> of colorizing to the status output:
> - general header (defaults to normal white)
> - updated but not committed (defaults to green)
> - changed but not updated (defaults to red)
> - untracked files (defaults to red)
> The idea is that red things indicate a potential mistake on the part of the
> user (e.g., forgetting to update a file, forgetting to git-add a file).
Perhaps the default values should not use the same color twice? I'd
suggest yellow for changed but not updated. But well, it's no problem
to change this in my config, I just find it a bit confusing to have
the same color for different things.
> Color support is controlled by status.color and status.color.*. There is no
> command line option, and the status.color variable is a simple boolean (no
> checking for tty output).
Is there any way to do isatty() from shell scripts?
^ permalink raw reply
* [PATCH 3/9] gitweb: Use undef for die_error to use default first (status) parameter value
From: Jakub Narebski @ 2006-08-05 10:56 UTC (permalink / raw)
To: git
In-Reply-To: <44d47813.36251c31.2553.3cf7@mx.gmail.com>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
For consistency.
gitweb/gitweb.perl | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 6feec28..9b9bf37 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -2320,7 +2320,7 @@ sub git_history {
sub git_search {
if (!defined $searchtext) {
- die_error("", "Text field empty.");
+ die_error(undef, "Text field empty.");
}
if (!defined $hash) {
$hash = git_read_head($project);
--
1.4.1.1
^ permalink raw reply related
* [PATCH 2/9] gitweb: die_error first (optional) parameter is HTTP status
From: Jakub Narebski @ 2006-08-05 10:56 UTC (permalink / raw)
To: git
In-Reply-To: <44d47813.36251c31.2553.3cf7@mx.gmail.com>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
gitweb/gitweb.perl | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 8b53fe6..6feec28 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -68,7 +68,7 @@ our $git_version = qx($GIT --version) =~
$projects_list ||= $projectroot;
if (! -d $git_temp) {
- mkdir($git_temp, 0700) || die_error("Couldn't mkdir $git_temp");
+ mkdir($git_temp, 0700) || die_error(undef, "Couldn't mkdir $git_temp");
}
# ======================================================================
@@ -1658,7 +1658,7 @@ sub git_blob_plain {
}
my $type = shift;
open my $fd, "-|", $GIT, "cat-file", "blob", $hash
- or die_error("Couldn't cat $file_name, $hash");
+ or die_error(undef, "Couldn't cat $file_name, $hash");
$type ||= git_blob_plain_mimetype($fd, $file_name);
--
1.4.1.1
^ permalink raw reply related
* [PATCH 4/9] gitweb: Don't undefine query parameter related variables before die_error
From: Jakub Narebski @ 2006-08-05 10:58 UTC (permalink / raw)
To: git
In-Reply-To: <44d47813.36251c31.2553.3cf7@mx.gmail.com>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
It would allow to include value of invalid parameter in error message
gitweb/gitweb.perl | 21 +++++----------------
1 files changed, 5 insertions(+), 16 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 9b9bf37..6f3f465 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -76,7 +76,6 @@ # input validation and dispatch
our $action = $cgi->param('a');
if (defined $action) {
if ($action =~ m/[^0-9a-zA-Z\.\-_]/) {
- undef $action;
die_error(undef, "Invalid action parameter.");
}
# action which does not check rest of parameters
@@ -89,16 +88,13 @@ if (defined $action) {
our $project = ($cgi->param('p') || $ENV{'PATH_INFO'});
if (defined $project) {
$project =~ s|^/||; $project =~ s|/$||;
- $project = validate_input($project);
- if (!defined($project)) {
+ if (!validate_input($project)) {
die_error(undef, "Invalid project parameter.");
}
if (!(-d "$projectroot/$project")) {
- undef $project;
die_error(undef, "No such directory.");
}
if (!(-e "$projectroot/$project/HEAD")) {
- undef $project;
die_error(undef, "No such project.");
}
$rss_link = "<link rel=\"alternate\" title=\"" . esc_param($project) . " log\" href=\"" .
@@ -111,32 +107,28 @@ if (defined $project) {
our $file_name = $cgi->param('f');
if (defined $file_name) {
- $file_name = validate_input($file_name);
- if (!defined($file_name)) {
+ if (!validate_input($file_name)) {
die_error(undef, "Invalid file parameter.");
}
}
our $hash = $cgi->param('h');
if (defined $hash) {
- $hash = validate_input($hash);
- if (!defined($hash)) {
+ if (!validate_input($hash)) {
die_error(undef, "Invalid hash parameter.");
}
}
our $hash_parent = $cgi->param('hp');
if (defined $hash_parent) {
- $hash_parent = validate_input($hash_parent);
- if (!defined($hash_parent)) {
+ if (!validate_input($hash_parent)) {
die_error(undef, "Invalid hash parent parameter.");
}
}
our $hash_base = $cgi->param('hb');
if (defined $hash_base) {
- $hash_base = validate_input($hash_base);
- if (!defined($hash_base)) {
+ if (!validate_input($hash_base)) {
die_error(undef, "Invalid hash base parameter.");
}
}
@@ -144,7 +136,6 @@ if (defined $hash_base) {
our $page = $cgi->param('pg');
if (defined $page) {
if ($page =~ m/[^0-9]$/) {
- undef $page;
die_error(undef, "Invalid page parameter.");
}
}
@@ -152,7 +143,6 @@ if (defined $page) {
our $searchtext = $cgi->param('s');
if (defined $searchtext) {
if ($searchtext =~ m/[^a-zA-Z0-9_\.\/\-\+\:\@ ]/) {
- undef $searchtext;
die_error(undef, "Invalid search parameter.");
}
$searchtext = quotemeta $searchtext;
@@ -182,7 +172,6 @@ my %actions = (
$action = 'summary' if (!defined($action));
if (!defined($actions{$action})) {
- undef $action;
die_error(undef, "Unknown action.");
}
$actions{$action}->();
--
1.4.1.1
^ permalink raw reply related
* [PATCH 5/9] gitweb: Cleanup and uniquify error messages
From: Jakub Narebski @ 2006-08-05 11:12 UTC (permalink / raw)
To: git
In-Reply-To: <44d47813.36251c31.2553.3cf7@mx.gmail.com>
Signed-off-by: Jakub Narebski <jnareb@gmail.com>
---
gitweb/gitweb.perl | 12 ++++++------
1 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 6f3f465..8773a8d 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -1464,7 +1464,7 @@ sub git_blame2 {
die_error(undef, "Permission denied.") if (!git_get_project_config_bool ('blame'));
die_error('404 Not Found', "File name not defined") if (!$file_name);
$hash_base ||= git_read_head($project);
- die_error(undef, "Reading commit failed") unless ($hash_base);
+ die_error(undef, "Couldn't find base commit.") unless ($hash_base);
my %co = git_read_commit($hash_base)
or die_error(undef, "Reading commit failed");
if (!defined $hash) {
@@ -1473,7 +1473,7 @@ sub git_blame2 {
}
$ftype = git_get_type($hash);
if ($ftype !~ "blob") {
- die_error("400 Bad Request", "object is not a blob");
+ die_error("400 Bad Request", "Object is not a blob");
}
open ($fd, "-|", $GIT, "blame", '-l', $file_name, $hash_base)
or die_error(undef, "Open git-blame failed.");
@@ -1520,9 +1520,9 @@ sub git_blame2 {
sub git_blame {
my $fd;
die_error('403 Permission denied', "Permission denied.") if (!git_get_project_config_bool ('blame'));
- die_error('404 Not Found', "What file will it be, master?") if (!$file_name);
+ die_error('404 Not Found', "File name not defined.") if (!$file_name);
$hash_base ||= git_read_head($project);
- die_error(undef, "Reading commit failed.") unless ($hash_base);
+ die_error(undef, "Couldn't find base commit.") unless ($hash_base);
my %co = git_read_commit($hash_base)
or die_error(undef, "Reading commit failed.");
if (!defined $hash) {
@@ -2113,7 +2113,7 @@ sub git_commitdiff {
open my $fd, "-|", $GIT, "diff-tree", '-r', $hash_parent, $hash
or die_error(undef, "Open git-diff-tree failed.");
my @difftree = map { chomp; $_ } <$fd>;
- close $fd or die_error(undef, "Reading diff-tree failed.");
+ close $fd or die_error(undef, "Reading git-diff-tree failed.");
# non-textual hash id's can be cached
my $expires;
@@ -2484,7 +2484,7 @@ sub git_rss {
open my $fd, "-|", $GIT, "rev-list", "--max-count=150", git_read_head($project)
or die_error(undef, "Open git-rev-list failed.");
my @revlist = map { chomp; $_ } <$fd>;
- close $fd or die_error(undef, "Reading rev-list failed.");
+ close $fd or die_error(undef, "Reading git-rev-list failed.");
print $cgi->header(-type => 'text/xml', -charset => 'utf-8');
print "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n".
"<rss version=\"2.0\" xmlns:content=\"http://purl.org/rss/1.0/modules/content/\">\n";
--
1.4.1.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox