* Re: [PATCH 1/6 (v4)] man page and technical discussion for rev-cache
From: Nick Edelen @ 2009-10-02 22:12 UTC (permalink / raw)
To: Nick Edelen, Junio C Hamano, Nicolas Pitre, Johannes Schindelin,
Sam Vilain
In-Reply-To: <op.uyuwkmv0tdk399@sirnot.private>
Before any code is introduced the full documentation is put forth. This
provides a man page for the porcelain, and a technical doc in technical/. The
latter describes the API, and discusses rev-cache's design, file format and
mechanics.
Signed-off-by: Nick Edelen <sirnot@gmail.com>
---
clean resend of patches. this set fixes a small bug in graft handling, and
tweaks test for compatability.
Documentation/git-rev-cache.txt | 190 ++++++++++
Documentation/technical/rev-cache.txt | 634 +++++++++++++++++++++++++++++++++
2 files changed, 824 insertions(+), 0 deletions(-)
diff --git a/Documentation/git-rev-cache.txt b/Documentation/git-rev-cache.txt
new file mode 100644
index 0000000..5a713ad
--- /dev/null
+++ b/Documentation/git-rev-cache.txt
@@ -0,0 +1,190 @@
+git-rev-cache(1)
+================
+
+NAME
+----
+git-rev-cache - Add, walk and maintain revision cache slices
+
+SYNOPSIS
+--------
+'git-rev-cache' COMMAND [options] [<commit>...]
+
+DESCRIPTION
+-----------
+The revision cache ('rev-cache') provides a mechanism for significantly
+speeding up revision traversals. It does this by creating an efficient
+database (cache) of commits, their related objects and topological relations.
+Independant of packs and the object store, this database is composed of
+rev-cache "slices" -- each a different file storing a given segment of commit
+history. To map commits to their respective slices, a single index file is
+kept for the rev-cache.
+
+'git-rev-cache' provides a front-end for the rev-cache mechanism, intended for
+updating and maintaining rev-cache slices in the current repository. New cache
+slice files can be 'add'ed, to keep the cache up-to-date; individual slices can
+be traversed; smaller slices can be 'fuse'd into a larger slice; and the
+rev-cache index can be regenerated.
+
+COMMANDS
+--------
+
+add
+~~~
+Add revisions to the cache by creating a new cache slice. Reads a revision
+list from the command line, formatted as: `START START ... \--not END END ...`
+
+Options
+^^^^^^^
+
+\--all::
+ Include all refs in the new cache slice, like the \--all option in
+ 'rev-list'.
+
+\--fresh/\--incremental::
+ Exclude everything already in the revision cache, analogous to
+ \--incremental in 'pack-objects'.
+
+\--stdin::
+ Read newline-seperated revisions from the standard input. Use \--not
+ to exclude commits, as on the command line.
+
+\--legs::
+ Ensure newly-generated cache slice has no partial ends. This means that
+ no commit has partially cached parents, in that all its parents are
+ cached or none of them are. 99.9% of users can ignore this command.
++
+\--legs will cause 'rev-cache' to expand potential slice end-points (creating
+"legs") until this condition is met, simplifying the cache slice structure.
+'rev-cache' itself does not care if a slice has legs or not, but the condition
+may reduce the required complexity of other applications that might use the
+revision cache.
+
+\--no-objects::
+ Non-commit objects are normally included along with the commit with
+ which they were introduced. This is obviously very benificial, but can
+ take longer in cache slice generation. Using this option will disable
+ non-commit object caching.
++
+\--no-objects is mainly intended for debugging or development purposes, but may
+find use in special situations (e.g. common traversal of only commits).
+
+Output
+^^^^^^
+
+On `stderr` 'add' outputs general information about the generated slice,
+including the number of objects and paths, and the start/end commits (prefix S
+indicates start, E an end). Through `stdout` it emits only the SHA-1 of the
+slice.
+
+walk
+~~~~
+Analogous to a slice-oriented 'rev-list', 'walk' will traverse a region in a
+particular cache slice. Interesting and uninteresting (delimited, as with
+'rev-list', with \--not) are specified on the command line, and output is the
+same as vanilla 'rev-list'.
+
+Options
+^^^^^^^
+
+\--objects::
+ Like 'rev-list', 'walk' will normally only list commits. Use this
+ option to list non-commit objects as well, if they are present in the
+ cache slice.
+
+Output
+^^^^^^
+
+'walk' will simply dump the contents of the output commit list, work list, and
+pending object array. The headers are outputed on `stderr`, the object hashes
+and names on `stdout`.
+
+fuse
+~~~~
+Merge several cache slices into a single large slice, like 'repack' for
+'rev-cache'. On each invocation of 'add' a new file ("slice") is added to the
+revision cache directory, and after several additions the directory may become
+populated with many, relatively small slices. Numerous smaller slices will
+yield poorer performance than a one or two large ones, because of the overhead
+of loading new slices into memory.
+
+Running 'fuse' every once in a while will solve this problem by coalescing all
+the cache slices into one larger slice. For very large projects, using
+\--ignore-size is advisable to prevent overly large cache slices. This can be
+set to run on garbage collection; see 'Automation' for more info.
+
+Note that 'fuse' uses the internal revision walker, so the options used in
+fusion override those of the cache slices upon which it operates. For example,
+if some slices were generated with \--no-objects, yet 'fuse' was performed with
+non-commit objects, the resulting slice would still contain objects but would
+take longer to generate.
+
+Options
+^^^^^^^
+
+\--all::
+ Normally fuse will only include everything that's already in the
+ revision cache. \--all tells it to start walking from the branch
+ heads, effectively a `add --all --fresh; fuse`
+ (pseudo-revcache-command).
+
+\--no-objects::
+ As in 'add', this option disables inclusion of non-commit objects. If
+ some cache slices do contain such objects, the information will be lost.
+
+\--ignore-size[=N]::
+ Do not merge cache slices of size >=N (be aware that slices must be
+ mapped to memory). N can have a suffix of "k" or "m", denoting N as
+ kilobytes and megabytes, respectively. If N is not provided 'fuse'
+ will default to a size specified in `revcache.ignoresize`, or ~25MB if
+ the config var is not set.
+
+Output
+^^^^^^
+
+This command prints the SHA-1 of the new slice on `stdout`, and information
+about its work on `stderr` -- specifically which files it's removing.
+
+Automation
+^^^^^^^^^^
+
+Set the git configuration variable `gc.revcache` to run 'fuse' on garbage
+collection. The arguments passed are `fuse \--all \--ignore-size`; i.e. 'gc'
+will keep everything cached into size-regulated slices.
+
+index
+~~~~~
+Regenerate the revision cache index. If the rev-cache index file associating
+objects with cache slices gets corrupted, lost, or otherwise becomes unusable,
+'index' will quickly regenerate the file. It's most likely that this won't be
+needed in every day use, as it is targeted towards debugging and development.
+
+alt
+~~~
+Create a cache slice pointer to another slice, identified by its full path:
+`fuse path/to/other/slice`
+
+This command is useful if you have several repositories sharing a common
+history. Although space requirements for rev-cache are slim anyway, you can in
+this situation reduce it further by using slice pointers, pointing to relavant
+slices in other repositories. Note that only one level of redirection is
+allowed, and the slice pointer will break if the original slice is removed.
+'fuse' will not touch slice pointers.
+
+NOTES
+-----
+In certain circumstances there may be some inconsistencies with object names
+between cached and non-cached walks. Specifically, if two objects in a commit
+tree have the same content (= same SHA-1); or if objects of the same SHA-1 are
+introduced independantly in parallel branches.
+
+In the first case rev-cache will use the name of the youngest file, while
+vanilla rev-list will return the name of the entry first encountered in walking
+the tree. The latter case is a result of rev-cache's internal topological
+ordering: the difference is the same between sorted and unsorted revision walks.
+
+See 'Discussion' for the underlying reasons for the discrepencies.
+
+DISCUSSION
+----------
+For an explanation of the API and its inner workings, see
+link:technical/rev-cache.txt[technical info on rev-cache].
diff --git a/Documentation/technical/rev-cache.txt b/Documentation/technical/rev-cache.txt
new file mode 100644
index 0000000..91fce8b
--- /dev/null
+++ b/Documentation/technical/rev-cache.txt
@@ -0,0 +1,634 @@
+rev-cache
+=========
+
+The revision cache API ('rev-cache') provides a method for efficiently storing
+and accessing commit branch sections. Such branch slices are defined by a
+series of start/top (interesting) and end/bottom (uninteresting) commits. Each
+slice contains information on commits in topological order. Recorded with each
+commit is:
+
+* All intra-slice topological relations, encoded into path "channels" (see
+ 'Mechanics' for full explanation).
+* Object meta-data: type, SHA-1, size, date (for commits).
+* Objects introduced by that commit, not present in the its cached parents.
+
+In addition to the API, basic structures are exported for the possibility of
+direct access.
+
+The API
+-------
+You can find the function prototypes in `revision.h`.
+
+Data Structures
+~~~~~~~~~~~~~~~
+The `rev_cache_info` struct holds all the options and flags for the API.
+
+----
+struct rev_cache_info {
+ /* generation flags */
+ unsigned objects : 1,
+ legs : 1,
+ make_index : 1,
+ fuse_me : 1;
+
+ /* index inclusion */
+ unsigned overwrite_all : 1;
+
+ /* traversal flags */
+ unsigned add_to_pending : 1;
+
+ /* fuse options */
+ unsigned int ignore_size;
+
+ /* reserved */
+ struct rev_cache_slice_map *maps,
+ *last_map;
+};
+----
+
+The fields:
+
+`objects`::
+ Add non-commit objects to slice.
+
+`legs`::
+ Ensure end/bottom commits have no children.
+
+`make_index`::
+ Integrate newly-made slice into index.
+
+`fuse_me`::
+ This is specified if a fuse is occuring, and slices are to be reused.
+ This option requires `maps` and `last_maps` to be initialized.
+
+`overwrite_all`::
+ When a cache slice is added to the index, sometimes overlap occures
+ between it and other slices. Normally, original index entries are kept
+ unless the new entry represents a start commit (older entries are more
+ likely to lead to greater in-slice traversals). This options overrides
+ that, and updates all entries of the new slice.
+
+`add_to_pending`::
+ Append unique non-commit objects to the `pending` object list in the
+ passed `rev_info` instance.
+
+`add_names`::
+ Include non-commit object names in the pending object entries if
+ `add_to_pending` is set.
+
+`ignore_size`::
+ If non-zero, ignore slices with size greater or equal to this during
+fusion.
+
+`maps`/`last_map`::
+ An array of slice mappings, indexed by their id in the slice index
+ header, to be re-used with `fuse_me`. `last_map` points to the last
+ mapping used, and should be initialized to 0.
+
+Functions
+~~~~~~~~~
+
+init_rev_cache
+^^^^^^^^^^^^^^
+----
+void init_rev_cache_info(
+ struct rev_cache_info *rci OUT
+)
+----
+
+Initialize `rci` to default options.
+
+make_cache_slice
+^^^^^^^^^^^^^^^^
+----
+int make_cache_slice(
+ struct rev_cache_info *rci IN,
+ struct rev_info *revs IN,
+ struct commit_list **starts IN/OUT,
+ struct commit_list **ends IN/OUT,
+ unsigned char *cache_sha1 OUT
+)
+----
+
+Create a cache slice based on either `revs` (if non-NULL) *or* the `starts` and
+`ends` lists. The actual list of start and end commits of the slice may be
+different from the parameters, based on what defines the branch segment, and
+this actual list is passed back through `starts` and `ends`.
+
+The cache slice is identified via a SHA-1 generated from the actual start/end
+commit lists. `cache_sha1`, if non-NULL, can recieve the cache slice name.
+`rci` is used to specify generation options, but can be NULL if you want
+`make_cache_slice` to fall back on defaults. Returns 0 on success, non-zero on
+failure.
+
+make_cache_index
+^^^^^^^^^^^^^^^^
+----
+int make_cache_index(
+ struct rev_cache_info *rci IN,
+ unsigned char *cache_sha1 IN,
+ int fd IN,
+ unsigned int size IN
+)
+----
+
+Add a slice to the rev-cache index. `cache_sha1` is the identity hash of the
+cache slice; `fd` is a file descriptor of the cache slice opened with
+read/write privileges (the slice is not actually modified); `size` is the size
+of the cache slice. Although there are currently no options for index
+updating, `rci` is a placeholder in case of future options. Note that this
+function is normally called by `make_cache_slice`. Returns 0 on success,
+non-zero on failure.
+
+open_cache_slice
+^^^^^^^^^^^^^^^^
+----
+int open_cache_slice(
+ unsigned char *sha1 IN,
+ int flags IN
+)
+----
+
+Returns a file descriptor to a cache slice described by `sha1` hash, using
+`flags` as the access mode. This will follow cache slice pointers to one level
+of indirection.
+
+get_cache_slice
+^^^^^^^^^^^^^^^
+----
+unsigned char *get_cache_slice(
+ struct commit *commit IN
+)
+----
+
+Given a commit object `get_cache_slice` will search the revision cache index
+and return, if found, the cache slice SHA-1.
+
+traverse_cache_slice
+^^^^^^^^^^^^^^^^^^^^
+----
+int traverse_cache_slice(
+ struct rev_info *revs IN/OUT,
+ unsigned char *cache_sha1 IN,
+ struct commit *commit IN,
+ unsigned long *date_so_far IN/OUT,
+ int *slop_so_far IN/OUT,
+ struct commit_list ***queue OUT,
+ struct commit_list **work IN/OUT
+)
+----
+
+Traverse a specified cache slice. An explanation of the each field:
+
+`revs`::
+ The revision walk instance. `traverse_cache_slice` uses this for
+ general options (e.g. which objects are included) and slice traversal
+ options (in the `rev_cache_info` field). If the `add_to_pending`
+ option is specified, non-commit objects are appended to the `pending`
+ object list field.
+
+`cache_sha1`::
+ SHA-1 identifying the cache slice to use. This can be taken directly
+ from `get_cache_slice`.
+
+`commit`::
+ The current commit object in the revision walk, i.e. the commit which
+ inspired this slice traversal. Although theoretically redundant in
+ view of the `work` list, this simplifies interaction with normal
+ revision walks, which pop commits from `work` before analyzing them.
+
+`date_so_far`::
+ The date of the oldest encountered interesting commit. Passing NULL
+ will let `traverse_cache_slice` use defaults.
+
+`slop_so_far`::
+ The `slop` value, a la revision.c. This is a counter used to determine
+ when to stop traversing, based on how many extra uninteresting commits
+ should be encountered. NULL will enable defaults, as above.
+
+`queue`::
+ Refers to a pointer to the head of a FIFO commit list, recieving the
+ commits we've seen and added.
+
+`work`::
+ A date-ordered list of commits that have yet to be processed (i.e. seen
+ but not added). Commits from here present in the slice are removed
+ (and, obviously, used as starting places for traversal), and any end
+ commits encountered are inserted.
+
+starts_from_slices
+^^^^^^^^^^^^^^^^^^
+----
+void starts_from_slices(
+ struct rev_info *revs OUT,
+ unsigned int flags IN,
+ unsigned char *which IN,
+ int n IN
+)
+----
+
+Will mark start-commits in certain rev-cache slices with `flag`, and added them
+to the pending list of `revs`. If `n` is zero, `starts_from_slices` will use
+all slices. Otherwise `which` will specify an *unseperated* list of cache
+SHA-1s to use (20 bytes each), and `n` will contain the number of slices (i.e.
+20 * `n` = size of `which`).
+
+fuse_cache_slices
+^^^^^^^^^^^^^^^^^
+----
+int fuse_cache_slices(
+ struct rev_cache_info *rci IN,
+ struct rev_info *revs IN
+)
+----
+
+Generate a slice based on `revs`, replacing all encountered slices with one
+(larger) slice. The `ignore_size` field in `rci`, if non-zero, will dictate
+which cache slice sizes to ignore in both traversal and replacement.
+
+regenerate_cache_index
+^^^^^^^^^^^^^^^^^^^^^^
+----
+int regenerate_cache_index(
+ struct rev_cache_info *rci IN
+)
+----
+
+Remake the revision cache index, including all the slices. Currently no
+options in `rci` exist for index (re)generation, but some may develop in the
+future.
+
+to/from_disked_rc_object/index_entry
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+----
+struct rc_object/index_entry *from_disked_rc_object/index_entry(
+ struct rc_object/index_entry_ondisk *src IN,
+ struct rc_object/index_entry *dst OUT
+)
+
+struct rc_object/index_entry_ondisk *to_disked_rc_object/index_entry(
+ struct rc_object/index_entry *src IN,
+ struct rc_object/index_entry_ondisk *dst OUT
+)
+----
+
+Functions to convert between the internal and storage (`_ondisk`) versions of
+object and index entry structures. These are necessary for direct access to
+the cache slices. If NULL is provided for `dst` a statically allocated
+structure is used, and a pointer to the struct is returned. Otherwise the
+functions return `dst`.
+
+Example Usage
+-------------
+
+A few examples to demonstrate usage:
+
+.Creating a slice
+----
+/* pretend you're a porcelain for rev-cache reading from the command line */
+struct rev_info revs;
+struct rev_cache_info rci;
+
+init_revisions(&revs, 0);
+init_rci(&rci);
+
+flags = 0;
+for (i = 1; i < argc; i++) {
+ if (!strcmp(argv[i], "--not"))
+ flags ^= UNINTERESTING;
+ else if(!strcmp(argv[i], "--fresh"))
+ starts_from_slices(&revs, UNINTERESTING, 0, 0);
+ else
+ handle_revision_arg(argv[i], &revs, flags, 1);
+}
+
+/* we want to explicitly set certain options */
+rci.objects = 0;
+
+if (!make_cache_slice(&rci, &revs, 0, 0, cache_sha1))
+ printf("made slice! it's called %s\n", sha1_to_hex(cache_sha1));
+----
+
+.Traversing a slice
+----
+/* let's say you're walking the tree with a 'work' list of current heads and a
+ * FILO output list 'out' */
+out = 0;
+outp = &out;
+
+while (work) {
+ struct commit *commit = pop_commit(&work);
+ struct object *object = &commit->object;
+ unsigned char *cache_sha1;
+
+ if (cache_sha1 = get_cache_slice(object->sha1)) {
+ /* note that this will instatiate any topo-relations
+ * as it goes */
+ if (traverse_cache_slice(&revs, cache_sha1,
+ commit, 0, 0, /* use defaults */
+ &outp, &work) < 0)
+ die("I'm overreacting to a non-fatal cache error");
+ } else {
+ struct commit_list *parents = commit->parents;
+
+ while (parents) {
+ struct commit *p = parents->item;
+ struct object *po = &p->object;
+
+ parents = parents->next;
+ if (po->flags & UNINTERESTING)
+ continue;
+
+ if (object->flags & UNINTERESTING)
+ po->flags |= UNINTERESTING;
+ else if (po->flags & SEEN)
+ continue;
+
+ if (!po->parsed)
+ parse_commit(p);
+ insert_by_date(p, &work);
+ }
+
+ if (object->flags & (SEEN | UNINTERESTING) == 0)
+ outp = &commit_list_insert(commit, outp)->next;
+ object->flags |= SEEN;
+ }
+}
+----
+
+Some Internals
+--------------
+For more advanced usage, the slice and index file(s) may be accessed directly.
+Relavant structures are availabe in `rev-cache.h`.
+
+File Formats
+~~~~~~~~~~~~
+
+Cache Slices
+^^^^^^^^^^^^
+A slice has a basic fixed-size header, followed by a certain number of object
+entries, then a NULL-seperated list of object names. Commits are sorted in
+topo-order, and each commit entry is followed by the objects added in that
+commit.
+
+----
+ -- +--------------------------------+
+header | object number, etc... |
+ -- +--------------------------------+
+commit | commit info |
+entry | path data |
+ +--------------------------------+
+ | tree/blob info |
+ +--------------------------------+
+ | tree/blob info |
+ +--------------------------------+
+ | ... |
+ -- +--------------------------------+
+commit | commit info |
+entry | path data |
+ +--------------------------------+
+ | tree/blob info |
+ +--------------------------------+
+ | ... |
+ -- +--------------------------------+
+... ...
+ -- +--------------------------------+
+name list | \0some_file_name\0 |
+(note +--------------------------------+
+preceeding | another_file\0 |
+null) ... |
+ +--------------------------------+
+----
+
+Here is the header:
+
+----
+struct rc_cache_slice_header {
+ char signature[8]; /* REVCACHE */
+ unsigned char version;
+ uint32_t ofs_objects;
+
+ uint32_t object_nr;
+ uint16_t path_nr;
+ uint32_t size;
+
+ unsigned char sha1[20];
+
+ uint32_t names_size;
+};
+----
+
+Explanations:
+
+`signature`::
+ The identifying signature of cache slice file. Always "REVCACHE".
+`version`::
+ The version number, currently 1.
+`ofs_objects`::
+ The byte offset at which the commit/object listing starts. Always
+ present at the 10th byte, regardless of file version.
+`object_nr`::
+ The total number of objects (commit + non-commit objects) present in
+ the slice.
+`path_nr`::
+ The total number of paths/channels used in encoding the topological
+ data. Note that paths are reused (see 'Mechanics'), so there will
+ never be more than a few hundred paths (if that) used.
+`size`::
+ The size of the slice *excluding* the name list. In other words, the
+ size of the portion mapped to memory.
+`sha1`::
+ The cache slice SHA-1.
+`names_size`::
+ The size of the name list. `size` + `names_size` = size of slice
+
+Revision Cache Index
+^^^^^^^^^^^^^^^^^^^^
+The index is a single file that associates SHA-1s with cache slices and file
+positions. It is somewhat similar to pack-file indexes, containing a fanout
+table and a list of index entries sorted by hash.
+
+----
+ -- +--------------------------------+
+header | object #, cache #, etc. |
+ -- +--------------------------------+
+sha1s of | SHA-1 |
+slices | ... |
+ -- +--------------------------------+
+fanout | fanout[0x00] |
+table ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ | fanout[0xff] |
+ -- +--------------------------------+
+index | SHA-1 of object |
+entries | index of cache slice SHA-1 |
+ | position in cache slice |
+ +--------------------------------+
+ | |
+ ...
+ +--------------------------------+
+----
+
+The header:
+
+----
+struct rc_index_header {
+ char signature[8]; /* REVINDEX */
+ unsigned char version;
+ uint32_t ofs_objects;
+
+ uint32_t object_nr;
+ unsigned char cache_nr;
+
+ uint32_t max_date;
+};
+----
+
+Explanations:
+
+`signature`::
+ Always "REVINDEX".
+`version`::
+ Version number, currently 1.
+`ofs_objects`::
+ Offset at which the entry objects begin. This is more obviously useful
+ in the index because the list of slice SHA-1s is variably-sized.
+`object_nr`::
+ Number of index entry objects present.
+`cache_nr`::
+ Number of cache slices to which the index maps, and hence the number of
+slice SHA-1s listed.
+`max_date`::
+ The oldest commit represented in the index. This is used to help speed
+up lookup times by knowing what range of commits we definitely don't have
+cached. Normal usage of 'rev-cache' would leave no "holes" in its coverage of
+commit history -- once a commit is cached, everything reachable from it should
+be cached as well. Most of the time refs are added to rev-cache simultaneous
+as well. This means that in most situations almost everything <= `max_date`
+will be cached.
+
+Mechanics
+~~~~~~~~~
+
+The most important part of rev-cache is its method of encoding topological
+relations. To ensure fluid traversal and reconstruction, commits are related
+through high-level "streams"/"channels" rather than individual
+interconnections. Intuitively, rev-cache stores history the way gitk shows it:
+commits strung up on lines, which interconnect at merges and branches.
+
+Each commit is associated to a given channel/path via a 'path id', and
+variable-length fields govern which paths (if any) are closed or opened at that
+object. This means that topo-data can be preserved in only a few bytes extra
+per object entry. Other information stored per entry is the sha-1 hash, type,
+date, size, name, and status in cache slice. Here is format of an object
+entry, both on-disk and in-memory:
+
+----
+struct object_entry {
+ unsigned type : 3;
+ unsigned is_end : 1;
+ unsigned is_start : 1;
+ unsigned uninteresting : 1;
+ unsigned include : 1;
+ unsigned flags : 1;
+ unsigned char sha1[20];
+
+ unsigned char merge_nr;
+ unsigned char split_nr;
+ unsigned size_size : 3;
+ unsigned name_size : 3;
+
+ uint32_t date;
+ uint16_t path;
+
+ /* merge paths */
+ /* split paths */
+ /* size */
+ /* name index */
+};
+----
+
+An explanation of each field:
+
+`type`::
+ Object type
+`is_end`::
+ The commit has some parents outside the cache slice (all if slice has
+ legs)
+`is_start`::
+ The commit has no children in cache slice
+`uninteresting`::
+ Run-time flag, used in traversal
+`include`::
+ Run-time flag, used in traversal (initialization)
+`flags`::
+ Currently unused, extra bit
+`sha1`::
+ Object SHA-1 hash
+
+`merge_nr`::
+ The number of paths the current channel diverges into; the current path
+ ends upon any merge.
+`split_nr`::
+ The number of paths this commit ends; used on both merging and
+ branching.
+`size_size`::
+ Number of bytes the object size takes up.
+`name_size`::
+ Number of bytes the name index takes up.
+
+`date`::
+ The date of the commit.
+`path`::
+ The path ID of the channel with which this commit is associated.
+
+merge paths::
+ The path IDs (16-bit) that are to be created. Overflow is not a
+ problem as path IDs are reused, leaving even complicated projects to
+ consume no more than a few hundred IDs.
+split paths::
+ The path IDs (16-bit) that are to be ended.
+size::
+ The size split into the minimum number of bytes. That is, 1-8 bytes
+ representing the size, least-significant byte first.
+name index::
+ An offset for the null-seperated, object name list at the end of the
+ cache slice. Also split into the minimum number of bytes.
+
+Each path ID refers to an index in a 'path array', which stores the current
+status (eg. active, interestingness) of each channel.
+
+Due to topo-relations and boundary tracking, all of a commit's parents must be
+encountered before the path is reallocated. This is achieved by using a
+counter system per merge: starting at the parent number, the counter is
+decremented as each parent is encountered (dictated by 'split paths'); at 0 the
+path is cleared.
+
+Boundary tracking is necessary because non-commits are stored relative to the
+commit in which they were introduced. If a series of commits is not included
+in the output, the last interesting commit must be parsed manually to ensure
+all objects are accounted for.
+
+To prevent list-objects from recursing into trees that we've already taken care
+of, the flag `FACE_VALUE` is introduced. An object with this flag is not
+explored (= "taken at face value"), significantly reducing I/O and processing
+time.
+
+Notes
+~~~~~
+
+Due to rev-cache's internal storage format, walking may lead to some
+discrepencies between cached and uncached repositories. Although noticeable to
+users directly calling rev-list, these are unused or corner cases and
+internally a non-issue.
+
+First note that rev-cache records commits in topological order. Large portions
+of commit history will already be sorted topologically in the revision walk,
+yielding a different output for unsorted calls to rev-list. A more obscure
+consquence occurs when two objects of the same SHA-1, but different name, are
+introduced seperately in parallel branches: different names might be shown for
+that object depending on which object entry was encountered first.
+
+A similar disparity arises when two objects of same SHA-1/different name are
+present in the same tree structure. rev-cache, walking objects as they were
+introduced, lists the youngest file's name; rev-list, walking the full trees
+each commit, shows the first file encountered.
--
tg: (5bbb081..) t/revcache/docs (depends on: t/revcache/integration)
^ permalink raw reply related
* Re: "Not currently on any branch"
From: Sean Estabrooks @ 2009-10-02 23:15 UTC (permalink / raw)
To: Alex Riesen; +Cc: Tim, git
In-Reply-To: <81b0412b0910021446nb07e7e9l465f588168297fe9@mail.gmail.com>
On Fri, 2 Oct 2009 23:46:53 +0200
Alex Riesen <raa.lkml@gmail.com> wrote:
> On Fri, Oct 2, 2009 at 22:08, Tim <timothyjwashington@yahoo.ca> wrote:
> > What's the most straightforward & cleanest way to merge my changes in the
> > headless branch to my 'ui-integration' branch?
>
> Assuming you use a Bourne shell:
>
> $ prev=$(git rev-parse HEAD)
> $ git checkout ui-integration && git merge $prev
>
>
You could also rely on the reflog to avoid the need to store off the
previous HEAD value, so just:
$ git checkout ui-integration && git merge HEAD@{1}
Sean
^ permalink raw reply
* Re: [PATCH] MSVC: fix build warnings
From: Michael Wookey @ 2009-10-02 23:28 UTC (permalink / raw)
To: Junio C Hamano, git
In-Reply-To: <7v7hvd4flb.fsf@alter.siamese.dyndns.org>
2009/10/3 Junio C Hamano <gitster@pobox.com>:
> Michael Wookey <michaelwookey@gmail.com> writes:
>
>> diff --git a/builtin-branch.c b/builtin-branch.c
>> index 9f57992..cf6a9ca 100644
>> --- a/builtin-branch.c
>> +++ b/builtin-branch.c
>> @@ -93,7 +93,7 @@ static const char *branch_get_color(enum color_branch ix)
>>
>> static int delete_branches(int argc, const char **argv, int force, int kinds)
>> {
>> - struct commit *rev, *head_rev = head_rev;
>
> I haven't tried, but the patch may break build with "gcc -Werror".
>
> This is a common and unfortunate idiom to tell the readers of the code
> that this initialization is unnecessary, gcc is not clever enough to
> notice and gives warnings, and we are squelching it, knowing what we are
> doing.
I can't build with -Werror on Ubuntu 9.04 (gcc 4.3.3) because of the following:
http://article.gmane.org/gmane.comp.version-control.git/127477
With the current git.rc2, I also get the following warnings:
builtin-mailinfo.c: In function 'handle_commit_msg':
builtin-mailinfo.c:789: warning: ignoring return value of
'ftruncate', declared with attribute warn_unused_result
It would be nice to get those warnings removed.
I just tried my patch with gcc 4.2.1 (Mac OSX 10.6) and there are a
few warnings that are generated because some of the variables have had
their initial values removed. I can send a V2 if you like, however
these variable were initialised that way for a reason and it might not
be sensible to clean them up in the way I was proposing.
What would be a good method of fixing these warnings now that we have
the ability to compile with MSVC? Explicitly initialising the
variables (to something sane) or should we start to introduce compiler
specific pragmas (ugly...) that aim to clean the various build
warnings? I just want to reduce (and eventually remove) all the build
noise when building using MSVC.
From what I have seen so far, building with MSVC spews out a lot of
warnings. I am building with MSVC in both the IDE and from a build
console via:
devenv git.sln /useenv /build "Debug|Win32"
If you compile using gcc with "-Wextra" you will see a similar amount
of build noise that gets generated. See the following for some
previous discussion:
http://article.gmane.org/gmane.comp.version-control.git/128967
^ permalink raw reply
* [PATCH/RFC 1/7] imap-send: use separate read and write fds
From: Erik Faye-Lund @ 2009-10-03 0:39 UTC (permalink / raw)
To: msysgit; +Cc: git
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
---
imap-send.c | 37 +++++++++++++++++++++++--------------
1 files changed, 23 insertions(+), 14 deletions(-)
diff --git a/imap-send.c b/imap-send.c
index 3847fd1..ba50961 100644
--- a/imap-send.c
+++ b/imap-send.c
@@ -154,7 +154,7 @@ struct imap_list {
};
struct imap_socket {
- int fd;
+ int fd[2];
SSL *ssl;
};
@@ -304,8 +304,12 @@ static int ssl_socket_connect(struct imap_socket *sock, int use_tls_only, int ve
ssl_socket_perror("SSL_new");
return -1;
}
- if (!SSL_set_fd(sock->ssl, sock->fd)) {
- ssl_socket_perror("SSL_set_fd");
+ if (!SSL_set_rfd(sock->ssl, sock->fd[0])) {
+ ssl_socket_perror("SSL_set_rfd");
+ return -1;
+ }
+ if (!SSL_set_wfd(sock->ssl, sock->fd[1])) {
+ ssl_socket_perror("SSL_set_wfd");
return -1;
}
@@ -327,11 +331,12 @@ static int socket_read(struct imap_socket *sock, char *buf, int len)
n = SSL_read(sock->ssl, buf, len);
else
#endif
- n = xread(sock->fd, buf, len);
+ n = xread(sock->fd[0], buf, len);
if (n <= 0) {
socket_perror("read", sock, n);
- close(sock->fd);
- sock->fd = -1;
+ close(sock->fd[0]);
+ close(sock->fd[1]);
+ sock->fd[0] = sock->fd[1] = -1;
}
return n;
}
@@ -344,11 +349,12 @@ static int socket_write(struct imap_socket *sock, const char *buf, int len)
n = SSL_write(sock->ssl, buf, len);
else
#endif
- n = write_in_full(sock->fd, buf, len);
+ n = write_in_full(sock->fd[1], buf, len);
if (n != len) {
socket_perror("write", sock, n);
- close(sock->fd);
- sock->fd = -1;
+ close(sock->fd[0]);
+ close(sock->fd[1]);
+ sock->fd[0] = sock->fd[1] = -1;
}
return n;
}
@@ -361,7 +367,8 @@ static void socket_shutdown(struct imap_socket *sock)
SSL_free(sock->ssl);
}
#endif
- close(sock->fd);
+ close(sock->fd[0]);
+ close(sock->fd[1]);
}
/* simple line buffering */
@@ -960,7 +967,7 @@ static void imap_close_server(struct imap_store *ictx)
{
struct imap *imap = ictx->imap;
- if (imap->buf.sock.fd != -1) {
+ if (imap->buf.sock.fd[0] != -1) {
imap_exec(ictx, NULL, "LOGOUT");
socket_shutdown(&imap->buf.sock);
}
@@ -988,7 +995,7 @@ static struct store *imap_open_store(struct imap_server_conf *srvc)
ctx = xcalloc(sizeof(*ctx), 1);
ctx->imap = imap = xcalloc(sizeof(*imap), 1);
- imap->buf.sock.fd = -1;
+ imap->buf.sock.fd[0] = imap->buf.sock.fd[1] = -1;
imap->in_progress_append = &imap->in_progress;
/* open connection to IMAP server */
@@ -1015,7 +1022,8 @@ static struct store *imap_open_store(struct imap_server_conf *srvc)
close(a[0]);
- imap->buf.sock.fd = a[1];
+ imap->buf.sock.fd[0] = a[1];
+ imap->buf.sock.fd[1] = dup(a[1]);
imap_info("ok\n");
} else {
@@ -1092,7 +1100,8 @@ static struct store *imap_open_store(struct imap_server_conf *srvc)
goto bail;
}
- imap->buf.sock.fd = s;
+ imap->buf.sock.fd[0] = s;
+ imap->buf.sock.fd[1] = dup(s);
if (srvc->use_ssl &&
ssl_socket_connect(&imap->buf.sock, 0, srvc->ssl_verify)) {
--
1.6.5.rc2.7.g4f8d3
^ permalink raw reply related
* [PATCH/RFC 2/7] imap-send: use run-command API for tunneling
From: Erik Faye-Lund @ 2009-10-03 0:39 UTC (permalink / raw)
To: msysgit; +Cc: git
In-Reply-To: <1254530385-2824-1-git-send-email-kusmabite@gmail.com>
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
---
imap-send.c | 37 ++++++++++++++++---------------------
1 files changed, 16 insertions(+), 21 deletions(-)
diff --git a/imap-send.c b/imap-send.c
index ba50961..55a663a 100644
--- a/imap-send.c
+++ b/imap-send.c
@@ -24,6 +24,7 @@
#include "cache.h"
#include "exec_cmd.h"
+#include "run-command.h"
#ifdef NO_OPENSSL
typedef void *SSL;
#endif
@@ -989,8 +990,7 @@ static struct store *imap_open_store(struct imap_server_conf *srvc)
struct imap_store *ctx;
struct imap *imap;
char *arg, *rsp;
- int s = -1, a[2], preauth;
- pid_t pid;
+ int s = -1, preauth;
ctx = xcalloc(sizeof(*ctx), 1);
@@ -1001,29 +1001,24 @@ static struct store *imap_open_store(struct imap_server_conf *srvc)
/* open connection to IMAP server */
if (srvc->tunnel) {
- imap_info("Starting tunnel '%s'... ", srvc->tunnel);
+ const char *argv[4];
+ struct child_process tunnel = {0};
- if (socketpair(PF_UNIX, SOCK_STREAM, 0, a)) {
- perror("socketpair");
- exit(1);
- }
+ imap_info("Starting tunnel '%s'... ", srvc->tunnel);
- pid = fork();
- if (pid < 0)
- _exit(127);
- if (!pid) {
- if (dup2(a[0], 0) == -1 || dup2(a[0], 1) == -1)
- _exit(127);
- close(a[0]);
- close(a[1]);
- execl("/bin/sh", "sh", "-c", srvc->tunnel, NULL);
- _exit(127);
- }
+ argv[0] = "/bin/sh";
+ argv[1] = "-c";
+ argv[2] = srvc->tunnel;
+ argv[3] = NULL;
- close(a[0]);
+ tunnel.argv = argv;
+ tunnel.in = -1;
+ tunnel.out = -1;
+ if (start_command(&tunnel))
+ die("cannot start proxy %s", argv[0]);
- imap->buf.sock.fd[0] = a[1];
- imap->buf.sock.fd[1] = dup(a[1]);
+ imap->buf.sock.fd[0] = tunnel.out;
+ imap->buf.sock.fd[1] = tunnel.in;
imap_info("ok\n");
} else {
--
1.6.5.rc2.7.g4f8d3
^ permalink raw reply related
* [PATCH/RFC 3/7] imap-send: fix compilation-error on Windows
From: Erik Faye-Lund @ 2009-10-03 0:39 UTC (permalink / raw)
To: msysgit; +Cc: git
In-Reply-To: <1254530385-2824-2-git-send-email-kusmabite@gmail.com>
mmsystem.h (included from windows.h) defines DRV_OK to 1. To avoid
an error due to DRV_OK redefenition, this patch undefines the old
definition (i.e the one from mmsystem.h) before defining DRV_OK.
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
---
imap-send.c | 4 ++++
1 files changed, 4 insertions(+), 0 deletions(-)
diff --git a/imap-send.c b/imap-send.c
index 55a663a..8338717 100644
--- a/imap-send.c
+++ b/imap-send.c
@@ -94,6 +94,10 @@ struct msg_data {
unsigned int crlf:1;
};
+#ifdef DRV_OK
+#undef DRV_OK
+#endif
+
#define DRV_OK 0
#define DRV_MSG_BAD -1
#define DRV_BOX_BAD -2
--
1.6.5.rc2.7.g4f8d3
^ permalink raw reply related
* [PATCH/RFC 4/7] imap-send: build imap-send on Windows
From: Erik Faye-Lund @ 2009-10-03 0:39 UTC (permalink / raw)
To: msysgit; +Cc: git
In-Reply-To: <1254530385-2824-3-git-send-email-kusmabite@gmail.com>
Since the POSIX-specific tunneling code has been replaced
by the run-command API (and a compile-error has been
cleaned away), we can now enable imap-send on Windows
builds.
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
---
Makefile | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/Makefile b/Makefile
index 12defd4..8ba789a 100644
--- a/Makefile
+++ b/Makefile
@@ -361,6 +361,7 @@ PROGRAMS += git-show-index$X
PROGRAMS += git-unpack-file$X
PROGRAMS += git-upload-pack$X
PROGRAMS += git-var$X
+PROGRAMS += git-imap-send$X
# List built-in command $C whose implementation cmd_$C() is not in
# builtin-$C.o but is linked in as part of some other command.
@@ -1056,7 +1057,6 @@ EXTLIBS += -lz
ifndef NO_POSIX_ONLY_PROGRAMS
PROGRAMS += git-daemon$X
- PROGRAMS += git-imap-send$X
endif
ifndef NO_OPENSSL
OPENSSL_LIBSSL = -lssl
--
1.6.5.rc2.7.g4f8d3
^ permalink raw reply related
* [PATCH/RFC 5/7] imap-send: provide fall-back random-source
From: Erik Faye-Lund @ 2009-10-03 0:39 UTC (permalink / raw)
To: msysgit; +Cc: git
In-Reply-To: <1254530385-2824-4-git-send-email-kusmabite@gmail.com>
Since some systems (at least Windows) does not have
/dev/random nor friends, we need another random-source.
This patch uses the C-runtime's rand()-function as a
poor-mans random-source.
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
---
imap-send.c | 17 ++++++++++-------
1 files changed, 10 insertions(+), 7 deletions(-)
diff --git a/imap-send.c b/imap-send.c
index 8338717..dda7b7f 100644
--- a/imap-send.c
+++ b/imap-send.c
@@ -511,14 +511,17 @@ static void arc4_init(void)
unsigned char j, si, dat[128];
if ((fd = open("/dev/urandom", O_RDONLY)) < 0 && (fd = open("/dev/random", O_RDONLY)) < 0) {
- fprintf(stderr, "Fatal: no random number source available.\n");
- exit(3);
- }
- if (read_in_full(fd, dat, 128) != 128) {
- fprintf(stderr, "Fatal: cannot read random number source.\n");
- exit(3);
+ /* poor-mans random-source */
+ srand(clock());
+ for (i = 0; i < 128; ++i)
+ dat[i] = rand() & 0xFF;
+ } else {
+ if (read_in_full(fd, dat, 128) != 128) {
+ fprintf(stderr, "Fatal: cannot read random number source.\n");
+ exit(3);
+ }
+ close(fd);
}
- close(fd);
for (i = 0; i < 256; i++)
rs.s[i] = i;
--
1.6.5.rc2.7.g4f8d3
^ permalink raw reply related
* [PATCH/RFC 6/7] mingw: wrap SSL_set_(w|r)fd to call _get_osfhandle
From: Erik Faye-Lund @ 2009-10-03 0:39 UTC (permalink / raw)
To: msysgit; +Cc: git
In-Reply-To: <1254530385-2824-5-git-send-email-kusmabite@gmail.com>
SSL_set_fd (and friends) expects a OS file handle on Windows, not
a file descriptor as on UNIX(-ish).
This patch makes the Windows version of SSL_set_fd behave like the
UNIX versions, by calling _get_osfhandle on it's input.
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
---
compat/mingw.h | 21 +++++++++++++++++++++
1 files changed, 21 insertions(+), 0 deletions(-)
diff --git a/compat/mingw.h b/compat/mingw.h
index 5b5258b..6907345 100644
--- a/compat/mingw.h
+++ b/compat/mingw.h
@@ -124,6 +124,27 @@ static inline int waitpid(pid_t pid, int *status, unsigned options)
return -1;
}
+#ifndef NO_OPENSSL
+#include <openssl/ssl.h>
+static inline int mingw_SSL_set_fd(SSL *ssl, int fd)
+{
+ return SSL_set_fd(ssl, _get_osfhandle(fd));
+}
+#define SSL_set_fd mingw_SSL_set_fd
+
+static inline int mingw_SSL_set_rfd(SSL *ssl, int fd)
+{
+ return SSL_set_rfd(ssl, _get_osfhandle(fd));
+}
+#define SSL_set_rfd mingw_SSL_set_rfd
+
+static inline int mingw_SSL_set_wfd(SSL *ssl, int fd)
+{
+ return SSL_set_wfd(ssl, _get_osfhandle(fd));
+}
+#define SSL_set_wfd mingw_SSL_set_wfd
+#endif
+
/*
* implementations of missing functions
*/
--
1.6.5.rc2.7.g4f8d3
^ permalink raw reply related
* [PATCH/RFC 7/7] mingw: enable OpenSSL
From: Erik Faye-Lund @ 2009-10-03 0:39 UTC (permalink / raw)
To: msysgit; +Cc: git
In-Reply-To: <1254530385-2824-6-git-send-email-kusmabite@gmail.com>
Since we have OpenSSL in msysgit now, enable it to support SSL
encryption for imap-send.
Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
---
Makefile | 1 -
1 files changed, 0 insertions(+), 1 deletions(-)
diff --git a/Makefile b/Makefile
index 8ba789a..8818f0f 100644
--- a/Makefile
+++ b/Makefile
@@ -933,7 +933,6 @@ else
ifneq (,$(findstring MINGW,$(uname_S)))
pathsep = ;
NO_PREAD = YesPlease
- NO_OPENSSL = YesPlease
NO_LIBGEN_H = YesPlease
NO_SYMLINK_HEAD = YesPlease
NO_IPV6 = YesPlease
--
1.6.5.rc2.7.g4f8d3
^ permalink raw reply related
* [PATCH] Fix '--relative-date'
From: Johan Sageryd @ 2009-10-03 4:20 UTC (permalink / raw)
To: git; +Cc: Johan Sageryd
This fixes '--relative-date' so that it does not give '0 year, 12 months', for the interval 360 <= diff < 365.
Signed-off-by: Johan Sageryd <j416@1616.se>
---
date.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/date.c b/date.c
index e9ee4aa..5d05ef6 100644
--- a/date.c
+++ b/date.c
@@ -123,7 +123,7 @@ const char *show_date_relative(unsigned long time, int tz,
return timebuf;
}
/* Say months for the past 12 months or so */
- if (diff < 360) {
+ if (diff < 365) {
snprintf(timebuf, timebuf_size, "%lu months ago", (diff + 15) / 30);
return timebuf;
}
--
1.6.4.4
^ permalink raw reply related
* Re: [PATCH 2/2] allow mangling short options which take integer arguments
From: Clemens Buchacher @ 2009-10-03 9:23 UTC (permalink / raw)
To: Jeff King; +Cc: Johannes Schindelin, Shawn O. Pearce, git
In-Reply-To: <20091002075012.GB27664@coredump.intra.peff.net>
On Fri, Oct 02, 2009 at 03:50:12AM -0400, Jeff King wrote:
> On Fri, Oct 02, 2009 at 09:43:17AM +0200, Clemens Buchacher wrote:
>
> > On Thu, Oct 01, 2009 at 11:55:03PM +0200, Johannes Schindelin wrote:
> >
> > > And this patch looks even more straight-forward than 1/2, _but_... what
> > > about cases where there are short options that are digits?
> >
> > Could you point me to one of those? I did not find any during my
> > non-exhaustive search. We should be able to handle them easily by adding
> > PARSE_OPT_MANY.
>
> The one that comes readily to mind is "git log -1", but that is actually
> parsed by the revision options parser, which doesn't use parseopt. But
> there are a few done by parseopt:
>
> $ git grep "OPT_.*'[0-9]'"
> archive.c: OPT__COMPR('1', &compression_level, "compress faster", 1),
> archive.c: OPT__COMPR_HIDDEN('2', &compression_level, 2),
> archive.c: OPT__COMPR_HIDDEN('3', &compression_level, 3),
> archive.c: OPT__COMPR_HIDDEN('4', &compression_level, 4),
> archive.c: OPT__COMPR_HIDDEN('5', &compression_level, 5),
> archive.c: OPT__COMPR_HIDDEN('6', &compression_level, 6),
> archive.c: OPT__COMPR_HIDDEN('7', &compression_level, 7),
> archive.c: OPT__COMPR_HIDDEN('8', &compression_level, 8),
> archive.c: OPT__COMPR('9', &compression_level, "compress better", 9),
> builtin-checkout.c: OPT_SET_INT('2', "ours", &opts.writeout_stage, "stage",
> builtin-checkout.c: OPT_SET_INT('3', "theirs", &opts.writeout_stage, "stage",
Those are not affected by this patch series. They can be cuddled with other
short options just like before, since they don't take arguments.
^ permalink raw reply
* Re: [PATCH/RFC 1/7] imap-send: use separate read and write fds
From: Jeff King @ 2009-10-03 9:40 UTC (permalink / raw)
To: Erik Faye-Lund; +Cc: msysgit, git
In-Reply-To: <1254530385-2824-1-git-send-email-kusmabite@gmail.com>
On Sat, Oct 03, 2009 at 12:39:39AM +0000, Erik Faye-Lund wrote:
> Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
Why? Given its presence in this series, I can only assume it has to do
with windows portability, but it would be helpful to give a little bit
of the reasoning in the commit message.
-Peff
^ permalink raw reply
* Re: [PATCH/RFC 5/7] imap-send: provide fall-back random-source
From: Jeff King @ 2009-10-03 9:58 UTC (permalink / raw)
To: Erik Faye-Lund; +Cc: msysgit, git
In-Reply-To: <1254530385-2824-5-git-send-email-kusmabite@gmail.com>
On Sat, Oct 03, 2009 at 12:39:43AM +0000, Erik Faye-Lund wrote:
> Since some systems (at least Windows) does not have
> /dev/random nor friends, we need another random-source.
>
> This patch uses the C-runtime's rand()-function as a
> poor-mans random-source.
Hmm. It looks like this arc4 RNG is used just for generating a unique
"X-TUID" header. Which seems to be used in isync (from which imap-send
is derived) to be to avoid duplicates in synchronization. But imap-send
doesn't actually use it for anything, as it just blindly pushes the
messages.
In other words, should all of this TUID code (and the arc4 code) simply
be ripped out?
-Peff
^ permalink raw reply
* Re: [PATCH] Fix '--relative-date'
From: Jeff King @ 2009-10-03 10:06 UTC (permalink / raw)
To: Johan Sageryd; +Cc: git
In-Reply-To: <1254543618-3772-1-git-send-email-j416@1616.se>
On Sat, Oct 03, 2009 at 01:20:18PM +0900, Johan Sageryd wrote:
> This fixes '--relative-date' so that it does not give '0 year, 12
> months', for the interval 360 <= diff < 365.
Thanks. I think this was a regression introduced recently when we fixed
the rounding on how years are printed (it used to just say "1 year"
because we were close, but now we always round down, so our boundary for
"1 year" must match exactly).
I've picked up your patch in my tree with the following test squashed
in:
diff --git a/t/t0006-date.sh b/t/t0006-date.sh
index a4d8fa8..75b02af 100755
--- a/t/t0006-date.sh
+++ b/t/t0006-date.sh
@@ -24,6 +24,7 @@ check_show 13000000 '5 months ago'
check_show 37500000 '1 year, 2 months ago'
check_show 55188000 '1 year, 9 months ago'
check_show 630000000 '20 years ago'
+check_show 31449600 '12 months ago'
check_parse() {
echo "$1 -> $2" >expect
-Peff
^ permalink raw reply related
* Automatic merge of local modifications in submodules?
From: Josef Wolf @ 2009-10-03 11:30 UTC (permalink / raw)
To: git
Hello,
I am trying to explore gti's submodule functionality. Since I'm used to
svn:externals, I am missing the automatic merge of local modifications.
So I'd like to do something like:
(cd submodule; git stash)
git submodule update
(cd submodule; git stash pop)
The problem with this command sequence is that it is not atomic. So if it
is automatically executed from a script (e.g cron or post-update-hook), it
will cause damage.
Any ideas how to properly auto-merge local modifications on submodule
update? git-submodule don't seem to have an option to do that.
^ permalink raw reply
* Re: [PATCH] Fix '--relative-date'
From: Johan Sageryd @ 2009-10-03 11:18 UTC (permalink / raw)
To: Jeff King; +Cc: git
In-Reply-To: <20091003100615.GC17873@coredump.intra.peff.net>
> I've picked up your patch in my tree with the following test squashed
> in:
>
> diff --git a/t/t0006-date.sh b/t/t0006-date.sh
> index a4d8fa8..75b02af 100755
> --- a/t/t0006-date.sh
> +++ b/t/t0006-date.sh
> @@ -24,6 +24,7 @@ check_show 13000000 '5 months ago'
> check_show 37500000 '1 year, 2 months ago'
> check_show 55188000 '1 year, 9 months ago'
> check_show 630000000 '20 years ago'
> +check_show 31449600 '12 months ago'
>
> check_parse() {
> echo "$1 -> $2" >expect
Thank you!
/Johan
^ permalink raw reply
* [PATCH] Reserve a slot for argv[0] in default_arg.
From: Petter Urkedal @ 2009-10-03 13:29 UTC (permalink / raw)
To: git; +Cc: Petter Urkedal
Setting "av" to one slot before the allocated "default_arg" array causes
glibc abort with "free(): invalid next size (normal)" in some
configurations (Gentoo, glibc-2.9_p20081201-r2, gcc-5.3.2 with PIE).
---
builtin-show-branch.c | 7 +++++--
1 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/builtin-show-branch.c b/builtin-show-branch.c
index 3510a86..3ab72b7 100644
--- a/builtin-show-branch.c
+++ b/builtin-show-branch.c
@@ -568,6 +568,9 @@ static int git_show_branch_config(const char *var, const char *value, void *cb)
if (default_alloc <= default_num + 1) {
default_alloc = default_alloc * 3 / 2 + 20;
default_arg = xrealloc(default_arg, sizeof *default_arg * default_alloc);
+ if (!default_num)
+ /* One unused position for argv[0]. */
+ default_arg[default_num++] = NULL;
}
default_arg[default_num++] = xstrdup(value);
default_arg[default_num] = NULL;
@@ -692,8 +695,8 @@ int cmd_show_branch(int ac, const char **av, const char *prefix)
/* If nothing is specified, try the default first */
if (ac == 1 && default_num) {
- ac = default_num + 1;
- av = default_arg - 1; /* ick; we would not address av[0] */
+ ac = default_num;
+ av = default_arg;
}
ac = parse_options(ac, av, prefix, builtin_show_branch_options,
--
1.6.4.4
^ permalink raw reply related
* Patches for libgit2
From: Hervé Cauwelier @ 2009-10-03 18:09 UTC (permalink / raw)
To: git
Hi,
Please find patches for libgit2. They allowed me to read the history of a
repository with only 75 patches but hacked by an history shortening.
Commits were both in a 5 GB pack file and loose objects. So I had to add
support for pack files. Hopefully, reading idx files was already there so I
followed the same model.
Patches can also be cloned at "git://git.hforge.org/libgit2.git".
There is a Python wrapper on the next side, forking from the existing
libgit2-python [1]. They bring the same functionnalities but the patches are
not slick yet.
[1] http://code.istique.net/gitweb/?p=libgit2-python.git;a=summary
One thing that would miss is unit tests, but I lack experience on testing C
code, and it would require having a small but exhaustive pack file with
different types of object, including ofs and ref deltas. Help on this part is
welcome.
Next step would be decoding raw objects to commits, trees, blobs and tags.
There is already the start of a commit structure. I'll be starting from that.
Regards,
Hervé
^ permalink raw reply
* [PATCH 1/6] Open the pack file and keep a map on it.
From: Hervé Cauwelier @ 2009-10-03 18:09 UTC (permalink / raw)
To: git; +Cc: Hervé Cauwelier
In-Reply-To: <1254593401-18801-1-git-send-email-herve@itaapy.com>
On the same model than the idx file.
Signed-off-by: Hervé Cauwelier <herve@itaapy.com>
---
src/odb.c | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
src/odb.h | 5 ++-
2 files changed, 68 insertions(+), 4 deletions(-)
diff --git a/src/odb.c b/src/odb.c
index 6d646a4..a562a19 100644
--- a/src/odb.c
+++ b/src/odb.c
@@ -64,6 +64,10 @@ struct git_pack {
/** Name of the pack file(s), without extension ("pack-abc"). */
char pack_name[GIT_PACK_NAME_MAX];
+
+ /** The .pack file, mapped into memory. */
+ git_file pack_fd;
+ git_map pack_map;
};
typedef struct git_pack git_pack;
@@ -782,7 +786,7 @@ static int pack_openidx(git_pack *p)
goto invalid_fail;
data = p->idx_map.data;
- if (decode32(&data[0]) == PACK_TOC) {
+ if (decode32(&data[0]) == IDX_TOC) {
switch (decode32(&data[1])) {
case 2:
if (pack_openidx_v2(p))
@@ -809,6 +813,59 @@ unlock_fail:
return GIT_ERROR;
}
+static int pack_openpack_map(git_pack *p)
+{
+ char pb[GIT_PATH_MAX];
+ off_t len;
+
+ if (git__fmt(pb, sizeof(pb), "%s/pack/%s.pack",
+ p->db->objects_dir,
+ p->pack_name) < 0)
+ return GIT_ERROR;
+
+ if ((p->pack_fd = gitfo_open(pb, O_RDONLY)) < 0)
+ return GIT_ERROR;
+
+ if ((len = gitfo_size(p->pack_fd)) < 0
+ || !git__is_sizet(len)
+ || gitfo_map_ro(&p->pack_map, p->pack_fd, 0, (size_t)len)) {
+ gitfo_close(p->pack_fd);
+ return GIT_ERROR;
+ }
+
+ return GIT_SUCCESS;
+}
+
+static int pack_openpack(git_pack *p)
+{
+ gitlck_lock(&p->lock);
+ if (p->invalid)
+ goto unlock_fail;
+ if (p->pack_fd < 0) {
+ uint32_t *data;
+
+ if (pack_openpack_map(p))
+ goto invalid_fail;
+ data = p->pack_map.data;
+
+ if (decode32(&data[0]) != PACK_TOC)
+ goto unmap_fail;
+ }
+ gitlck_unlock(&p->lock);
+ return GIT_SUCCESS;
+
+unmap_fail:
+ gitfo_free_map(&p->pack_map);
+
+invalid_fail:
+ p->invalid = 1;
+ p->pack_fd = -1;
+
+unlock_fail:
+ gitlck_unlock(&p->lock);
+ return GIT_ERROR;
+}
+
static void pack_decidx(git_pack *p)
{
gitlck_lock(&p->lock);
@@ -830,6 +887,11 @@ static void pack_dec(git_pack *p)
gitfo_close(p->idx_fd);
free(p->im_fanout);
}
+ if (p->pack_fd >= 0) {
+ gitfo_free_map(&p->pack_map);
+ gitfo_close(p->pack_fd);
+ p->pack_fd = -1;
+ }
gitlck_free(&p->lock);
free(p);
@@ -861,6 +923,7 @@ static git_pack *alloc_pack(const char *pack_name)
gitlck_init(&p->lock);
strcpy(p->pack_name, pack_name);
p->refcnt = 1;
+ p->pack_fd = -1;
return p;
}
@@ -895,7 +958,7 @@ static int scan_one_pack(void *state, char *name)
r->next = *ret;
*ret = r;
- return 0;
+ return GIT_SUCCESS;
}
static git_packlist* scan_packs(git_odb *db)
diff --git a/src/odb.h b/src/odb.h
index 2f205b2..0311d78 100644
--- a/src/odb.h
+++ b/src/odb.h
@@ -11,9 +11,10 @@
* uint32_t *fanout = ... the file data at offset 0 ...
* ntohl(fanout[0]) < ntohl(fanout[1])
*
- * The value chosen here for PACK_TOC is such that the above
+ * The value chosen here for IDX_TOC is such that the above
* cannot be true for an idx v1 file.
*/
-#define PACK_TOC 0xff744f63 /* -1tOc */
+#define IDX_TOC 0xff744f63 /* -1tOc */
+#define PACK_TOC 0x5041434b /* PACK */
#endif
--
1.6.4.4
^ permalink raw reply related
* [PATCH 2/6] Read the base offset or name of delta objects
From: Hervé Cauwelier @ 2009-10-03 18:09 UTC (permalink / raw)
To: git; +Cc: Hervé Cauwelier
In-Reply-To: <1254593401-18801-2-git-send-email-herve@itaapy.com>
Signed-off-by: Hervé Cauwelier <herve@itaapy.com>
---
src/cc-compat.h | 3 +++
src/odb.c | 28 ++++++++++++++++++++++++++--
2 files changed, 29 insertions(+), 2 deletions(-)
diff --git a/src/cc-compat.h b/src/cc-compat.h
index 8997caa..8dd6774 100644
--- a/src/cc-compat.h
+++ b/src/cc-compat.h
@@ -30,6 +30,9 @@
# define GIT_TYPEOF(x)
#endif
+#define bitsizeof(x) (CHAR_BIT * sizeof(x))
+#define MSB(x, bits) ((x) & GIT_TYPEOF(x)(~0ULL << (bitsizeof(x) - (bits))))
+
/*
* Does our compiler/platform support the C99 <inttypes.h> and
* <stdint.h> header files. (C99 requires that <inttypes.h>
diff --git a/src/odb.c b/src/odb.c
index a562a19..3cfc932 100644
--- a/src/odb.c
+++ b/src/odb.c
@@ -97,8 +97,10 @@ struct git_odb {
};
typedef struct { /* object header data */
- git_otype type; /* object type */
- size_t size; /* object size */
+ git_otype type; /* object type */
+ size_t size; /* object size */
+ off_t base_offset; /* delta base offset (GIT_OBJ_OFS_DELTA) */
+ git_oid base_name; /* delta base name (GIT_OBJ_REF_DELTA) */
} obj_hdr;
static struct {
@@ -238,6 +240,7 @@ static size_t get_binary_object_header(obj_hdr *hdr, gitfo_buf *obj)
unsigned char c;
unsigned char *data = obj->data;
size_t shift, size, used = 0;
+ off_t base_offset;
if (obj->len == 0)
return 0;
@@ -258,6 +261,27 @@ static size_t get_binary_object_header(obj_hdr *hdr, gitfo_buf *obj)
}
hdr->size = size;
+ hdr->base_offset = 0;
+ hdr->base_name.id[0] = '\0';
+
+ if (hdr->type == GIT_OBJ_OFS_DELTA) {
+ c = data[used++];
+ base_offset = c & 127;
+ while (c & 128) {
+ base_offset++;
+ if (!base_offset || MSB(base_offset, 7))
+ return 0; /* overflow */
+ c = data[used++];
+ base_offset = (base_offset << 7) + (c & 127);
+ }
+ assert(base_offset > 0);
+ hdr->base_offset = base_offset;
+ }
+ else if (hdr->type == GIT_OBJ_REF_DELTA) {
+ git_oid_mkraw(&hdr->base_name, data + used);
+ used += 20;
+ }
+
return used;
}
--
1.6.4.4
^ permalink raw reply related
* [PATCH 4/6] Inflate an object from a pack file
From: Hervé Cauwelier @ 2009-10-03 18:09 UTC (permalink / raw)
To: git; +Cc: Hervé Cauwelier
In-Reply-To: <1254593401-18801-4-git-send-email-herve@itaapy.com>
Support delta objects too.
Signed-off-by: Hervé Cauwelier <herve@itaapy.com>
---
src/odb.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
1 files changed, 68 insertions(+), 1 deletions(-)
diff --git a/src/odb.c b/src/odb.c
index 89ee1f2..bc26cf7 100644
--- a/src/odb.c
+++ b/src/odb.c
@@ -28,6 +28,7 @@
#include "git/zlib.h"
#include "fileops.h"
#include "hash.h"
+#include "delta-apply.h"
#include "odb.h"
#define GIT_PACK_NAME_MAX (5 + 40 + 1)
@@ -232,7 +233,7 @@ static int is_zlib_compressed_data(unsigned char *data)
unsigned int w;
w = ((unsigned int)(data[0]) << 8) + data[1];
- return data[0] == 0x78 && !(w %31);
+ return data[0] == 0x78 && !(w % 31);
}
static size_t get_binary_object_header(obj_hdr *hdr, gitfo_buf *obj)
@@ -1192,6 +1193,72 @@ int git_odb__read_loose(git_obj *out, git_odb *db, const git_oid *id)
return GIT_SUCCESS;
}
+static int inflate_pack_obj(git_obj *out, git_pack *p, off_t offset)
+{
+ obj_hdr hdr;
+ gitfo_buf buf;
+ size_t used;
+ void *data;
+ git_obj base;
+
+ /* Cast the map to a gitfo_buf */
+ buf.data = (unsigned char *)p->pack_map.data + offset;
+ buf.len = p->pack_map.len - offset;
+
+ /*
+ * Read the object header, which is an (uncompressed)
+ * binary encoding of the object type and size.
+ */
+ if (!(used = get_binary_object_header(&hdr, &buf)))
+ return GIT_ERROR;
+
+ /*
+ * Read the object data as a zlib compressed data
+ */
+ buf.data += used;
+ buf.len -= used;
+ assert(is_zlib_compressed_data(buf.data));
+
+ if (!(data = git__malloc(hdr.size + 1)))
+ return GIT_ERROR;
+ if (inflate_buffer(buf.data, buf.len, data, hdr.size))
+ goto inflate_fail;
+
+ switch (hdr.type) {
+ case GIT_OBJ_COMMIT:
+ case GIT_OBJ_TREE:
+ case GIT_OBJ_BLOB:
+ case GIT_OBJ_TAG:
+ out->data = data;
+ out->len = hdr.size;
+ out->type = hdr.type;
+ return GIT_SUCCESS;
+ case GIT_OBJ_OFS_DELTA:
+ offset -= hdr.base_offset;
+ if (inflate_pack_obj(&base, p, offset))
+ goto inflate_fail;
+ if (git__delta_apply(out, base.data, base.len, data, hdr.size))
+ goto inflate_fail;
+ out->type = base.type;
+ return GIT_SUCCESS;
+ case GIT_OBJ_REF_DELTA:
+ if (p->idx_search(&offset, p, &hdr.base_name))
+ goto inflate_fail;
+ if (inflate_pack_obj(&base, p, offset))
+ goto inflate_fail;
+ if (git__delta_apply(out, base.data, base.len, data, hdr.size))
+ goto inflate_fail;
+ out->type = base.type;
+ return GIT_SUCCESS;
+ default:
+ goto inflate_fail;
+ }
+
+inflate_fail:
+ free(data);
+ return GIT_ERROR;
+}
+
static int read_packed(git_obj *out, git_pack *p, const git_oid *id)
{
off_t pos;
--
1.6.4.4
^ permalink raw reply related
* [PATCH 3/6] Allow zlib to read a pack buffer longer than the actual data
From: Hervé Cauwelier @ 2009-10-03 18:09 UTC (permalink / raw)
To: git; +Cc: Hervé Cauwelier
In-Reply-To: <1254593401-18801-3-git-send-email-herve@itaapy.com>
As we don't know where the compressed data end, only the size of the
uncompressed data.
Signed-off-by: Hervé Cauwelier <herve@itaapy.com>
---
src/odb.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/src/odb.c b/src/odb.c
index 3cfc932..89ee1f2 100644
--- a/src/odb.c
+++ b/src/odb.c
@@ -432,7 +432,7 @@ static int inflate_buffer(void *in, size_t inlen, void *out, size_t outlen)
inflateEnd(&zs);
- if ((status != Z_STREAM_END) || (zs.avail_in != 0))
+ if (status != Z_STREAM_END)
return GIT_ERROR;
if (zs.total_out != outlen)
--
1.6.4.4
^ permalink raw reply related
* [PATCH 6/6] Read an object from a pack file
From: Hervé Cauwelier @ 2009-10-03 18:10 UTC (permalink / raw)
To: git; +Cc: Hervé Cauwelier
In-Reply-To: <1254593401-18801-6-git-send-email-herve@itaapy.com>
Signed-off-by: Hervé Cauwelier <herve@itaapy.com>
---
src/odb.c | 11 +++++++----
1 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/src/odb.c b/src/odb.c
index dd3c142..a712d71 100644
--- a/src/odb.c
+++ b/src/odb.c
@@ -1263,13 +1263,16 @@ static int read_packed(git_obj *out, git_pack *p, const git_oid *id)
if (pack_openidx(p))
return GIT_ERROR;
+ if (pack_openpack(p)) {
+ pack_decidx(p);
+ return GIT_ERROR;
+ }
res = p->idx_search(&pos, p, id);
pack_decidx(p);
+ assert(pos < p->pack_map.len);
- if (!res) {
- /* TODO unpack object at pos */
- res = GIT_ERROR;
- }
+ if (res == GIT_SUCCESS)
+ return inflate_pack_obj(out, p, pos);
return res;
}
--
1.6.4.4
^ permalink raw reply related
* [PATCH 5/6] This assertion is valid for both loose and packed objects
From: Hervé Cauwelier @ 2009-10-03 18:10 UTC (permalink / raw)
To: git; +Cc: Hervé Cauwelier
In-Reply-To: <1254593401-18801-5-git-send-email-herve@itaapy.com>
Signed-off-by: Hervé Cauwelier <herve@itaapy.com>
---
src/odb.c | 9 +++------
1 files changed, 3 insertions(+), 6 deletions(-)
diff --git a/src/odb.c b/src/odb.c
index bc26cf7..dd3c142 100644
--- a/src/odb.c
+++ b/src/odb.c
@@ -1149,11 +1149,10 @@ void git_odb_close(git_odb *db)
free(db);
}
-int git_odb_read(
- git_obj *out,
- git_odb *db,
- const git_oid *id)
+int git_odb_read(git_obj *out, git_odb *db, const git_oid *id)
{
+ assert(out && db && id);
+
attempt:
if (!git_odb__read_packed(out, db, id))
return GIT_SUCCESS;
@@ -1171,8 +1170,6 @@ int git_odb__read_loose(git_obj *out, git_odb *db, const git_oid *id)
char file[GIT_PATH_MAX];
gitfo_buf obj = GITFO_BUF_INIT;
- assert(out && db && id);
-
out->data = NULL;
out->len = 0;
out->type = GIT_OBJ_BAD;
--
1.6.4.4
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox