* Re: GIT on MinGW problem
From: Marius Storm-Olsen @ 2007-05-29 11:45 UTC (permalink / raw)
To: Johannes Sixt; +Cc: git
In-Reply-To: <465C064F.B9CE9379@eudaptics.com>
[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]
Johannes Sixt said the following on 29.05.2007 12:54:
> Han-Wen Nienhuys wrote:
>> Johannes Sixt escreveu:
>>> * git without an correct git subcommand should list 20 or so
>>> commands, but it doesn't. The list is just empty.
>> there was a problem in generate cmd list, (I have sort in /bin/
>> ). I recommend to add
>
> Strange. Here, MSYS aliases /usr to /, hence /usr/bin/sort is the
> same as /bin/sort.
>
> (For the curious ones: The MinGW port has to replace occurrences of
> 'sort' by '/usr/bin/sort', otherwise Windows's 'sort' would be
> picked up in shell scripts, because the latter usually comes first
> in %PATH%^W$PATH. Same for 'find'.)
I get that here too, no matter what I set the mount point to be, and
without the fstab file at all.
Also, the /bin/gitk.bat file should rather be
@"%~d0%~p0wish84.exe" "%~d0%~p0gitk" %*
than the current hardcoded path. (Probably won't work with
command.com, but who uses that for development nowadays anyways, right ;-)
--
.marius
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 187 bytes --]
^ permalink raw reply
* Re: What's new in git-gui.git
From: Petr Baudis @ 2007-05-29 11:18 UTC (permalink / raw)
To: Shawn O. Pearce; +Cc: git
In-Reply-To: <20070529041108.GG7044@spearce.org>
On Tue, May 29, 2007 at 06:11:08AM CEST, Shawn O. Pearce wrote:
> Apparently the Linux folks have this new fangled inotify thing
> that is also good at telling applications about changed files.
> It seems the hg folks are using it to speed up "hg status" by
> having a background daemon keep track of which files are dirty,
> and which aren't.
How well does this play with NFS? I'm working at two places and at both
have my ~ on NFS and sometimes touch the working tree from various
machines. Would the inotify-based approach silently fail here? Or does
inotify just refuse to work on NFS? Quick google search suggests the
former, which is more than ok for informative purposes but big problem
if you actually _rely_ on the inotify to keep your index up-to-date.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Ever try. Ever fail. No matter. // Try again. Fail again. Fail better.
-- Samuel Beckett
^ permalink raw reply
* Re: [PATCH 00/15] git-note: A mechanisim for providing free-form after-the-fact annotations on commits
From: Johannes Schindelin @ 2007-05-29 11:12 UTC (permalink / raw)
To: Andy Parkins
Cc: git, Linus Torvalds, Shawn O. Pearce, Junio C Hamano,
Johan Herland
In-Reply-To: <200705291204.10606.andyparkins@gmail.com>
Hi,
On Tue, 29 May 2007, Andy Parkins wrote:
> On Tuesday 2007 May 29, Linus Torvalds wrote:
>
> > So you'd have to have a separate database. You could do it with a
> > separate index file (or mixing it up with the "index v4" and doing it
> > with a single index file that also contains normal objects), but the
> > point is, it's going to be a real separate database.
>
> This is an off-the-wall suggestion; but why not use the object database
> a bit more strongly than just for storing the notes?
I alluded to the same yesterday.
However, just before waking up, I had the same idea as you, using a
branch.
> So where a normal tree has
>
> 100644 blob 156e952df8603c72532bcda95ddcd3bcb16ec5fd somefile
>
> A "notes" tree would have
>
> 100644 blob 156e952df8603c72532bcda95ddcd3bcb16ec5fd $SHA1_OF_TARGET_COMMIT1
> 100644 blob 97e08b0ab483146cb8fff31138eaa828c24ac84f $SHA1_OF_TARGET_COMMIT2
Actually, just throw in one fan-out stage, and you should be fine,
performance-wise.
And what is best: you could merge notes from somebody else easily,
_exactly_ because it is a proper branch now.
Ciao,
Dscho
^ permalink raw reply
* Re: [PATCH 00/15] git-note: A mechanisim for providing free-form after-the-fact annotations on commits
From: Andy Parkins @ 2007-05-29 11:04 UTC (permalink / raw)
To: git; +Cc: Linus Torvalds, Shawn O. Pearce, Junio C Hamano, Johan Herland
In-Reply-To: <alpine.LFD.0.98.0705282008400.26602@woody.linux-foundation.org>
On Tuesday 2007 May 29, Linus Torvalds wrote:
> So you'd have to have a separate database. You could do it with a separate
> index file (or mixing it up with the "index v4" and doing it with a single
> index file that also contains normal objects), but the point is, it's
> going to be a real separate database.
This is an off-the-wall suggestion; but why not use the object database a bit
more strongly than just for storing the notes? The original method was to
use the filesystem to store refs/notes/$SHA1, which was essentially annotated
tags. The object database has been designed for storing thousands of
hierarchically organised files, so why not use that instead of the refs/
tree?
That is to say, make a "private" branch, independent of any real repository
branches. I'll call it "notes". Then each commit in that branch represents
a change to a note (or multiple notes). The tree attached to each commit
provides the reverse lookup that's wanted, and also maps to the object
itself. So where a normal tree has
100644 blob 156e952df8603c72532bcda95ddcd3bcb16ec5fd somefile
A "notes" tree would have
100644 blob 156e952df8603c72532bcda95ddcd3bcb16ec5fd $SHA1_OF_TARGET_COMMIT1
100644 blob 97e08b0ab483146cb8fff31138eaa828c24ac84f $SHA1_OF_TARGET_COMMIT2
All the "notes" tools could now use the standard git tools to look up a note.
The notes would be version controlled (which is impressive in itself), and
reverse lookup is the same as path-limited git-rev-list.
The tree objects are (I guess) stored alphabetically, so the reverse lookup
could be done with a binary search. If a user really wanted to, they could
check out the "notes" branch and edit the files directly.
Madness? I'm sure.
Andy
--
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com
^ permalink raw reply
* Re: What's new in git-gui.git
From: Johannes Schindelin @ 2007-05-29 10:59 UTC (permalink / raw)
To: Shawn O. Pearce; +Cc: git
In-Reply-To: <20070529041108.GG7044@spearce.org>
Hi,
On Tue, 29 May 2007, Shawn O. Pearce wrote:
> Apparently the Linux folks have this new fangled inotify thing that is
> also good at telling applications about changed files. It seems the hg
> folks are using it to speed up "hg status" by having a background daemon
> keep track of which files are dirty, and which aren't.
An SCM which runs a background process? Please, no.
Next thing is, you have a daemon running which permanently scans your
source code for copynotsoright violations.
Ciao,
Dscho
^ permalink raw reply
* Re: GIT on MinGW problem
From: Johannes Sixt @ 2007-05-29 10:54 UTC (permalink / raw)
To: git
In-Reply-To: <f3a2ke$9s7$1@sea.gmane.org>
Han-Wen Nienhuys wrote:
> Johannes Sixt escreveu:
> > * git without an correct git subcommand should list 20 or so commands,
> > but it doesn't. The list is just empty.
>
> there was a problem in generate cmd list, (I have sort in /bin/ ). I
> recommend to add
Strange. Here, MSYS aliases /usr to /, hence /usr/bin/sort is the same
as /bin/sort.
(For the curious ones: The MinGW port has to replace occurrences of
'sort' by '/usr/bin/sort', otherwise Windows's 'sort' would be picked up
in shell scripts, because the latter usually comes first in
%PATH%^W$PATH. Same for 'find'.)
-- Hannes
^ permalink raw reply
* What's in git.git (stable)
From: Junio C Hamano @ 2007-05-29 10:12 UTC (permalink / raw)
To: git
In-Reply-To: <7vmyzv1acz.fsf@assigned-by-dhcp.cox.net>
Time for 1.5.2.1 perhaps.
The second batch of random changes are in 'master' now. This is
a rather large-ish looking one. Handle it with care.
----------------------------------------------------------------
* The 'maint' branch has these fixes since the last announcement.
Andy Parkins (2):
Fix mishandling of $Id$ expanded in the repository copy in convert.c
Add test case for $Id$ expanded in the repository
Carlos Rica (1):
fix memory leak in parse_object when check_sha1_signature fails
Eric Wong (1):
git-svn: avoid md5 calculation entirely if SVN doesn't provide one
Frank Lichtenheld (3):
cvsserver: Correct inetd.conf example in asciidoc documentation
cvsserver: Note that CVS_SERVER can also be specified as method variable
cvsserver: Fix some typos in asciidoc documentation
Jakub Narebski (3):
Documentation: Clean up links in GIT Glossary
Replace the last 'dircache's by 'index'
Documentation: Add definition of "evil merge" to GIT Glossary
James Bowes (1):
Documentation: fix git-config.xml generation
James Y Knight (1):
Fix git-svn to handle svn not reporting the md5sum of a file, and test.
Jeff King (2):
git-am: use printf instead of echo on user-supplied strings
More echo "$user_message" fixes.
Johan Herland (1):
Fix stupid typo in lookup_tag()
Jonas Fonseca (1):
Update bash completion to ignore some more plumbing commands
Junio C Hamano (3):
name-rev: tolerate clock skew in committer dates
git-commit: use printf '%s\n' instead of echo on user-supplied strings
Add tests for the last two fixes.
Nguyễn Thái Ngọc Duy (1):
Makefile: Remove git-fsck and git-verify-pack from PROGRAMS
Shawn O. Pearce (12):
git-gui: Tighten internal pattern match for lib/ directory
Refactor fast-import branch creation from existing commit
Fix possible coredump with fast-import --import-marks
Hide the plumbing diff-{files,index,tree} from bash completion
Teach bash completion about git-shortlog
Remove a duplicate --not option in bash completion
Update bash completion header documentation
Teach bash completion about 'git remote update'
Teach bash completion about recent log long options
Update bash completion for git-config options
Correct key bindings to Control-<foo>
git-gui: Guess our share/git-gui/lib path at runtime if possible
Simon Hausmann (2):
fast-import: Fix uninitialized variable
fast-import: Fix crash when referencing already existing objects
Steffen Prohaska (1):
user-manual: fixed typo in example
* The 'master' branch has these since the last announcement
in addition to the above.
Alex Riesen (6):
Add run_command_v_opt_cd: chdir into a directory before exec
Add ability to specify environment extension to run_command
Allow environment variables to be unset in the processes started by run_command
Verbose connect messages to show the IP addresses used
Add another verbosity level to git-fetch
Add a configuration option to control diffstat after merge
Dana L. How (7):
Alter sha1close() 3rd argument to request flush only
git-repack --max-pack-size: new file statics and code restructuring
git-repack --max-pack-size: write_{object,one}() respect pack limit
git-repack --max-pack-size: split packs as asked by write_{object,one}()
git-repack --max-pack-size: add option parsing to enable feature
pack-objects: clarification & option checks for --max-pack-size
Ensure git-repack -a -d --max-pack-size=N deletes correct packs
Daniel Barkalow (5):
Move remote parsing into a library file out of builtin-push.
Move refspec parser from connect.c and cache.h to remote.{c,h}
Add handlers for fetch-side configuration of remotes.
Update local tracking refs when pushing
Move refspec pattern matching to match_refs().
Fernando J. Pereda (1):
Teach mailsplit about Maildir's
Frank Lichtenheld (5):
t9400: Add test cases for config file handling
t9400: Add some more cvs update tests
t9400: Add some basic pserver tests
t9400: Work around CVS' deficiencies
cvsserver: Handle 'cvs login'
Junio C Hamano (4):
pack-objects: pass fullname down to add_object_entry()
Teach "delta" attribute to pack-objects.
builtin-pack-objects: remove unnecessary code for no-delta
mailsplit: fix for more than one input files
Linus Torvalds (2):
Make "git gc" pack all refs by default
Make the pack-refs interfaces usable from outside
Mark Levedahl (1):
gitweb.perl - Optionally send archives as .zip files
Nicolas Pitre (3):
fixes to output of git-verify-pack -v
improve delta long block matching with big files
update diff-delta.c copyright
Robin Rosenberg (1):
Add option to cvs update before export
Shawn O. Pearce (1):
Allow contrib new-workdir to link into bare repositories
^ permalink raw reply
* What's cooking in git.git (topics)
From: Junio C Hamano @ 2007-05-29 10:11 UTC (permalink / raw)
To: git
In-Reply-To: <7vodkb1adr.fsf@assigned-by-dhcp.cox.net>
Tonight's 'next' is broken in that it does not seem to be able
to do "git cat-file -t aba170cdb4874b72dd619e6f7bbc13c33295f83".
If you add "1" to the end, it becomes the commit v1.5.2^0.
Bisecting shows "Lazily open pack index files on demand" is the
culprit, so I've reverted it locally (and made sure things
starts working again), but I haven't got around to pushing out
the results yet. I won't, until tomorrow evening.
I'm a bit tired and it is getting late, so I won't comment on
each of the series. I am seriously considering about merging
Pasky's applypatch removal soon to 'master'.
----------------------------------------------------------------
Here are the topics that have been cooking. Commits prefixed
with '-' are only in 'pu' while commits prefixed with '+' are
in 'next'. The topics list the commits in reverse chronological
order.
* sp/pack (Tue May 29 02:49:08 2007 -0700) 4 commits
. Revert "Lazily open pack index files on demand"
+ Attempt to delay prepare_alt_odb during get_sha1
+ Micro-optimize prepare_alt_odb
+ Lazily open pack index files on demand
* pb/am (Thu May 24 19:25:25 2007 -0700) 2 commits
+ Remove git-applypatch
+ git-applymbox: Remove command
* mk/pack (Mon May 28 23:20:59 2007 +0200) 3 commits
+ builtin-pack-object: cache small deltas
+ git-pack-objects: cache small deltas between big objects
+ builtin-pack-objects: don't fail, if delta is not possible
* lh/submodules (Sat May 26 15:56:40 2007 +0200) 1 commit
+ Add git-submodule command
* dh/repack (Fri May 25 14:40:24 2007 -0700) 1 commit
- Enhance unpack-objects for live repo and large objects
* jc/blame (Fri Apr 20 16:25:50 2007 -0700) 4 commits
- blame: show log as it goes
- git-blame: optimize get_origin() from linear search to hash-
lookup.
- git-blame: pass "struct scoreboard *" pointers around.
- blame: lift structure definitions up
* jc/diff (Mon Dec 25 01:08:50 2006 -0800) 2 commits
- test-para: combined diff between HEAD, index and working tree.
- para-walk: walk n trees, index and working tree in parallel
^ permalink raw reply
* Re: git-svn dcommit failure
From: Florian Weimer @ 2007-05-29 9:35 UTC (permalink / raw)
To: git
In-Reply-To: <87veecat2k.fsf@mid.deneb.enyo.de>
* Florian Weimer:
> This with git-svn 1.5.1.4-2 on something that is quite close to Debian
> etch.
Same issue with 1.5.2~rc3-2.
^ permalink raw reply
* Re: sbuild (or whatever) integration
From: Florian Weimer @ 2007-05-29 9:26 UTC (permalink / raw)
To: Jan Hudec; +Cc: git
In-Reply-To: <20070521181505.GA5082@efreet.light.src>
* Jan Hudec:
> IMHO generic tools won't help you much, because every other command is Debian
> specific.
Yeah, and I mainly wanted to reuse process, not code. Thanks for your
comments.
^ permalink raw reply
* Re: [PATCH 00/15] git-note: A mechanisim for providing free-form after-the-fact annotations on commits
From: Johan Herland @ 2007-05-29 9:23 UTC (permalink / raw)
To: Jeff King; +Cc: Junio C Hamano, Linus Torvalds, git
In-Reply-To: <20070529082245.GA15788@coredump.intra.peff.net>
On Tuesday 29 May 2007, Jeff King wrote:
> On Tue, May 29, 2007 at 09:06:29AM +0200, Johan Herland wrote:
>
> > 1. Keep a file, ".git/reverse_tagmap_sorted" with one entry of the form
> > "pointee pointer" per line. The file is sorted on "pointee", so we can
> > easily do the radix-256-fan-out-followed-by-binary-search trick that
> > Linus mentioned in another thread. This should hopefully make lookup
> > fairly cheap. BTW, if there is a similar "pointee pointer"-type format
> > already being used in git, I'd be happy to use that instead. I looked
>
> I did a similar thing (though rather than having "lines" at all, they
> were fixed-length pairs of binary sha1 hashes) to implement a negative
> delta cache (which turned out to be a stupid idea, but the
> implementation worked):
>
> http://www.gelato.unsw.edu.au/archives/git/0606/23229.html
Cool. It won't give a big-Oh improvement, but it might give a worthwile
boost anyway. I guess it's the usual tradeoff between speed and
maintainability/transparency/readability.
> > 2. Keep another file, ".git/reverse_tagmap_unsorted" in front of (1).
> > This file has exactly the same format, minus the sorting. It exists just
> > to make insertion cheap. Once this file reaches a certain size (i.e.
> > when trawling it on lookup becomes slightly painful), we shuffle the
> > entries into the sorted file (this would happen automatically on
> > insertion of an entry, and should _not_ have to be triggered by 'git-gc'
> > etc.).
>
> The implementation I mentioned above collects several "to be inserted"
> entries (in memory) and then merge sorts the two lists into a new file.
> So it was fast in terms of comparisons, but it involved writing O(n)
> entries, which is probably bad for creating a single note.
>
> > Of course, if we think insertion directly into (1) will never be too
> > expensive, we can drop (2) altogether.
>
> You can always find the right spot, memmove everything down one slot,
> and insert. But that means:
> - the cost of insertion will be proportional to the number of items,
> whereas using an unsorted journal you get to amortize that cost over
> many insertions
> - if you update in place, you have to lock the db for reading while
> you are moving around. If you are always either appending to the
> journal or merging into a new file, you can avoid this.
Yep, I'm thinking (2) will be worth it in the end. O(#notes) per note
insertion doesn't sound appealing at all, especially not if we're
expecting a handful of notes per commit.
Have fun!
...Johan
--
Johan Herland, <johan@herland.net>
www.herland.net
^ permalink raw reply
* git-svn dcommit failure
From: Florian Weimer @ 2007-05-29 9:16 UTC (permalink / raw)
To: git
I suddenly can't run dcommit on some of my working copies. The change
I'm trying to push is pretty innocuous: just a single-line change (no
new file or directory). dcommit fails with: "Network connection
closed unexpectedly: Connection closed unexpectedly at
/usr/bin/git-svn line 405". I've looked at the data that is received
over SSH, and it looks pretty regular to me (that is, no interspersed
error message or something like that).
Is there any option to generate more debug output?
This with git-svn 1.5.1.4-2 on something that is quite close to Debian
etch.
^ permalink raw reply
* Re: [PATCH 00/15] git-note: A mechanisim for providing free-form after-the-fact annotations on commits
From: Jeff King @ 2007-05-29 8:22 UTC (permalink / raw)
To: Johan Herland; +Cc: Junio C Hamano, Linus Torvalds, git
In-Reply-To: <200705290906.29328.johan@herland.net>
On Tue, May 29, 2007 at 09:06:29AM +0200, Johan Herland wrote:
> 1. Keep a file, ".git/reverse_tagmap_sorted" with one entry of the form
> "pointee pointer" per line. The file is sorted on "pointee", so we can
> easily do the radix-256-fan-out-followed-by-binary-search trick that
> Linus mentioned in another thread. This should hopefully make lookup
> fairly cheap. BTW, if there is a similar "pointee pointer"-type format
> already being used in git, I'd be happy to use that instead. I looked
I did a similar thing (though rather than having "lines" at all, they
were fixed-length pairs of binary sha1 hashes) to implement a negative
delta cache (which turned out to be a stupid idea, but the
implementation worked):
http://www.gelato.unsw.edu.au/archives/git/0606/23229.html
> 2. Keep another file, ".git/reverse_tagmap_unsorted" in front of (1).
> This file has exactly the same format, minus the sorting. It exists just
> to make insertion cheap. Once this file reaches a certain size (i.e.
> when trawling it on lookup becomes slightly painful), we shuffle the
> entries into the sorted file (this would happen automatically on
> insertion of an entry, and should _not_ have to be triggered by 'git-gc'
> etc.).
The implementation I mentioned above collects several "to be inserted"
entries (in memory) and then merge sorts the two lists into a new file.
So it was fast in terms of comparisons, but it involved writing O(n)
entries, which is probably bad for creating a single note.
> Of course, if we think insertion directly into (1) will never be too
> expensive, we can drop (2) altogether.
You can always find the right spot, memmove everything down one slot,
and insert. But that means:
- the cost of insertion will be proportional to the number of items,
whereas using an unsorted journal you get to amortize that cost over
many insertions
- if you update in place, you have to lock the db for reading while
you are moving around. If you are always either appending to the
journal or merging into a new file, you can avoid this.
-Peff
^ permalink raw reply
* Re: What's new in git-gui.git
From: Alex Riesen @ 2007-05-29 8:12 UTC (permalink / raw)
To: Shawn O. Pearce; +Cc: git
In-Reply-To: <20070529041108.GG7044@spearce.org>
On 5/29/07, Shawn O. Pearce <spearce@spearce.org> wrote:
> Apparently the Linux folks have this new fangled inotify thing
... since july 2005
^ permalink raw reply
* Re: What's new in git-gui.git
From: Johannes Sixt @ 2007-05-29 7:46 UTC (permalink / raw)
To: git
In-Reply-To: <20070528224949.GC7044@spearce.org>
"Shawn O. Pearce" wrote:
> - Preserve author during amend
... and author date - at least if no additional changes were staged.
(Quite often I just want to tweak the commit message, mostly after
dissecting a topic branch.)
-- Hannes
^ permalink raw reply
* Re: GIT on MinGW - No symbolic links support
From: Johannes Sixt @ 2007-05-29 7:11 UTC (permalink / raw)
To: git
In-Reply-To: <Pine.LNX.4.64.0705280054140.4648@racer.site>
Johannes Schindelin wrote:
>
> Hi,
>
> On Sun, 27 May 2007, Aaron Gray wrote:
>
> > Bit of a dampener on GIT on MinGW :-
> >
> > $ git clone git://git.kernel.org/pub/scm/git/git.git
> > Initialized empty Git repository in C:/MSYS/src/git/.git/
> > error: git-checkout-index: unable to create symlink RelNotes (Function
> > not implemented)
> >
> > No Symbolic links !
> >
> > There are symbolic links provided by Windows by SFU (Services For Unix)
> > apparently.
>
> Does not work on FAT. Has lots of problems.
>
> That's why Johannes Sixt pushed for core.symlinks, and got it. So maybe
> the templates should set core.symlinks=false?
This setting should be placed into $(sysconfigdir)/gitconfig. I'll hack
up something in the next days. The "challenge" is that $(sysconfigdir)
must not be hardcoded (because the installation location is not known
until runtime). A similar thing I have already done with
$(template_dir).
-- Hannes
^ permalink raw reply
* [RFC] super indexes to span multiple packfiles
From: Shawn O. Pearce @ 2007-05-29 7:16 UTC (permalink / raw)
To: git; +Cc: Dana How, Nicolas Pitre
Dana talking about some sort of global index that spans all packfiles
got me thinking. Could we build a "super index" that covered all
known packfiles at the time it was created, but that doesn't really
hurt us when the packfiles are repacked? Or that doesn't have to
be repacked every single time a packfile is added to the repository.
This patch introduces a new command `git super-index` which reads
all known pack-*.idx files and merges them together into a single
super-index.sdx file, located in $GIT_OBJECT_DIRECTORY/pack.
At runtime we load the first super-index.sdx file we come across
during our packfile scanning. Using a single super-index works
because the super-index.sdx should contain data for this repository's
packs, and for all of its alternate's packs.
A super-index holds a list of packfile names that it will match at
the start of its data file. A super-index can only be used for
the pack(s) mentioned there. At runtime we try to lazily match
these pack SHA-1 names up to the struct packed_git.sha1 field,
giving us the struct packed_git* for direct index access.
The super-index does not hold the object offsets; instead it holds
an array of percentage shifts within the index fan-out ranges.
The shifts are normalized into the range 1-65535.
The basic idea is:
1) Use the first byte of the input SHA-1 to index into the
fan-out table of the super-index.sdx.
2) Use the next sdx_prefix bytes of the SHA-1 to binary search
through that region of the sdx file. Since we are working
only with the abbreviated prefix the memcmp() has less data to
wade through. We've also eliminated the common first byte,
thanks to the fan-out table.
3) The matching record in the sdx table holds sdx_packs uint16_t
values. The indexes in that list corresponding to the index
of the pack SHA-1 names in the header of the sdx file; hence
the first uint16_t goes to the first packfile, the second to
the second, etc. I call these "shifts". I don't know why.
4) If the shift value for a pack is 0 then that pack does not
contain any objects that start with that prefix. So a 0 shift
means we do not need to scan the corresponding pack's .idx file
for an object.
5) If the shift is non-zero then there is at least one object in
the corresponding packfile starting with the prefix and further
searching in the specific packfile is required. The shift is
actually the percentage within the range of [lo, hi) in the
pack's .idx of where the middle of that prefix's block appears.
We use this as a seed value for mi, rather than (lo+hi)/2, as it
gets us much much closer to the interesting SHA-1s in the .idx.
6) We binary search the .idx in a traditional way, once we have
selected a reasonable guess for the starting position.
7) If a packfile cannot be found at runtime (e.g. it has been
repacked away) we treat it's shift as though it were 0;
the object is not in that packfile (since the packfile does
not exist).
8) If the sdx cannot give us the object, we fallback to standard
packfile scanning.
In the single packfile case (everything repacked into one) this
is not faster; its actually slightly slower. With a handful of
smaller recent packfiles (such as immediately after a git-fetch)
it breaks even with the stock code. I haven't tested it yet with
a high number of packfiles (e.g. 20). I suspect it won't gain us
a lot up there either...
So in short this shouldn't be applied, because its not any faster,
and is sometimes slower. But I'm tossing it out here for discussion.
I'm also not documenting the new super-index command line program,
because I don't think this should be applied. ;-)
----
.gitignore | 1 +
Makefile | 1 +
builtin-pack-objects.c | 2 +-
builtin-super-index.c | 226 ++++++++++++++++++++++++++++++++++++++++++++++++
builtin.h | 1 +
cache.h | 2 +-
git.c | 1 +
pack-check.c | 4 +-
pack.h | 17 ++++
sha1_file.c | 208 ++++++++++++++++++++++++++++++++++++--------
10 files changed, 423 insertions(+), 40 deletions(-)
create mode 100644 builtin-super-index.c
diff --git a/.gitignore b/.gitignore
index 4dc0c39..3cd1b2c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -126,6 +126,7 @@ git-ssh-push
git-ssh-upload
git-status
git-stripspace
+git-super-index
git-svn
git-svnimport
git-symbolic-ref
diff --git a/Makefile b/Makefile
index 29243c6..26ac766 100644
--- a/Makefile
+++ b/Makefile
@@ -371,6 +371,7 @@ BUILTIN_OBJS = \
builtin-shortlog.o \
builtin-show-branch.o \
builtin-stripspace.o \
+ builtin-super-index.o \
builtin-symbolic-ref.o \
builtin-tar-tree.o \
builtin-unpack-objects.o \
diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index d165f10..3b49336 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -756,7 +756,7 @@ static int add_object_entry(const unsigned char *sha1, enum object_type type,
}
for (p = packed_git; p; p = p->next) {
- off_t offset = find_pack_entry_one(sha1, p);
+ off_t offset = find_pack_entry_one(sha1, p, 0);
if (offset) {
if (!found_pack) {
found_offset = offset;
diff --git a/builtin-super-index.c b/builtin-super-index.c
new file mode 100644
index 0000000..9f47760
--- /dev/null
+++ b/builtin-super-index.c
@@ -0,0 +1,226 @@
+#include "cache.h"
+#include "pack.h"
+#include "csum-file.h"
+
+struct sdx_entry {
+ unsigned char prefix[20];
+ uint16_t *splits;
+};
+
+static unsigned pack_cnt;
+static unsigned entry_cnt;
+static unsigned avail_cnt;
+static unsigned prefix_len = 5;
+static uint32_t *current;
+static struct packed_git **packs;
+static struct sdx_entry *entries;
+
+static unsigned split_avail;
+static uint16_t *split_free;
+
+static void select_packs()
+{
+ struct packed_git *p;
+ unsigned i;
+
+ prepare_packed_git();
+ for (p = packed_git; p; p = p->next)
+ pack_cnt++;
+
+ packs = xmalloc(pack_cnt * sizeof(packs[0]));
+ current = xcalloc(pack_cnt, sizeof(current[0]));
+ for (p = packed_git, i = 0; p; p = p->next, i++) {
+ packs[i] = p;
+ open_pack_index(p);
+ }
+}
+
+static void compute_pack(struct sdx_entry *ent, unsigned pack_idx)
+{
+ struct packed_git *p = packs[pack_idx];
+ const uint32_t *level1_ofs = p->index_data;
+ uint32_t hi, lo, mi, n = current[pack_idx];
+
+ /* Determine the end of the range that matches */
+ while (n < p->num_objects) {
+ const unsigned char *c = nth_packed_object_sha1(p, n);
+ if (memcmp(c, ent->prefix, prefix_len) > 0)
+ break;
+ n++;
+ }
+ if (n == current[pack_idx])
+ return;
+
+ /* Determine the position within the fan-out subrange */
+ if (p->index_version > 1)
+ level1_ofs += 2;
+ hi = ntohl(level1_ofs[*ent->prefix]);
+ lo = *ent->prefix ? ntohl(level1_ofs[*ent->prefix - 1]) : 0;
+ mi = (current[pack_idx] + n) / 2;
+ ent->splits[pack_idx] = (mi - lo) * 65535 / (hi - lo) + 1;
+
+ current[pack_idx] = n;
+}
+
+static int compute_next_entry()
+{
+ unsigned i;
+ const unsigned char *min = NULL;
+ struct sdx_entry *ent;
+
+ /* locate the minimum hash */
+ for (i = 0; i < pack_cnt; i++) {
+ struct packed_git *p = packs[i];
+ uint32_t n = current[i];
+ if (n < p->num_objects) {
+ const unsigned char *c = nth_packed_object_sha1(p, n);
+ if (!min || memcmp(c, min, prefix_len) < 0)
+ min = c;
+ }
+ }
+ if (!min)
+ return 0;
+
+ if (entry_cnt == avail_cnt) {
+ avail_cnt = avail_cnt * 3 / 2 + 128;
+ entries = xrealloc(entries, avail_cnt * sizeof(entries[0]));
+ }
+ if (split_avail < pack_cnt) {
+ split_avail = pack_cnt * (avail_cnt - entry_cnt);
+ split_free = xcalloc(split_avail, sizeof(split_free[0]));
+ }
+
+ ent = &entries[entry_cnt++];
+ hashcpy(ent->prefix, min);
+
+ ent->splits = split_free;
+ split_free += pack_cnt;
+ split_avail -= pack_cnt;
+
+ /* determine which packs match this entry */
+ for (i = 0; i < pack_cnt; i++)
+ compute_pack(ent, i);
+
+ return 1;
+}
+
+static int read_only(const char *path)
+{
+ mode_t mode = umask(0);
+
+ umask(mode);
+ mode = 0444 & ~mode;
+ if (chmod(path, mode))
+ return -1;
+ return adjust_shared_perm(path);
+}
+
+static void write_super_index()
+{
+ char tmpname[PATH_MAX];
+ struct sha1file *f;
+ uint32_t i, array[256];
+ int fd;
+ struct pack_sdx_header hdr;
+ struct sdx_entry *curr = entries, *last = entries + entry_cnt;
+
+ snprintf(tmpname, sizeof(tmpname), "%s/%s",
+ get_object_directory(),
+ "tmp_sdx_XXXXXX");
+ fd = mkstemp(tmpname);
+ if (fd < 0)
+ die("unable to create %s: %s\n", tmpname, strerror(errno));
+ f = sha1fd(fd, tmpname);
+
+ /* Our file header, for once we plan ahead! */
+ memset(&hdr, 0, sizeof(hdr));
+ hdr.sdx_signature = htonl(PACK_SDX_SIGNATURE);
+ hdr.sdx_version = htonl(1);
+ hdr.sdx_packs = htons(pack_cnt);
+ hdr.sdx_prefix = prefix_len - 1;
+ sha1write(f, &hdr, sizeof(hdr));
+
+ /* Immediately after is the pack SHA-1 names */
+ for (i = 0; i < pack_cnt; i++)
+ sha1write(f, packs[i]->sha1, sizeof(packs[i]->sha1));
+
+ /*
+ * Write the first-level table (the list is sorted,
+ * but we use a 256-entry lookup to be able to avoid
+ * having to do eight extra binary search iterations).
+ */
+ for (i = 0; i < 256; i++) {
+ while (curr < last && curr->prefix[0] == i)
+ curr++;
+ array[i] = htonl(curr - entries);
+ }
+ sha1write(f, array, 256 * 4);
+
+ /* Write the entries */
+ for (curr = entries; curr < last; curr++) {
+ sha1write(f, curr->prefix + 1, prefix_len - 1);
+ sha1write(f, curr->splits, pack_cnt);
+ }
+
+ sha1close(f, NULL, 1);
+ if (read_only(tmpname))
+ die("cannot make file readable: %s", strerror(errno));
+
+ if (move_temp_to_file(tmpname,
+ mkpath("%s/pack/super-index.sdx", get_object_directory())))
+ die("cannot save super-index.sdx");
+}
+
+static void show_super_index()
+{
+ unsigned i, j, multi_pack = 0;
+
+ for (i = 0; i < pack_cnt; i++)
+ printf("pack %s\n", sha1_to_hex(packs[i]->sha1));
+
+ for (i = 0; i < entry_cnt; i++) {
+ struct sdx_entry *ent = &entries[i];
+ char *prefix_str = sha1_to_hex(ent->prefix);
+ unsigned matches = 0;
+
+ prefix_str[2 * prefix_len] = 0;
+
+ printf("%s:", prefix_str);
+ for (j = 0; j < pack_cnt; j++) {
+ if (ent->splits[j]) {
+ matches++;
+ printf(" %3u", (int)ent->splits[j]);
+ } else
+ printf(" ----");
+ }
+ printf("\n");
+
+ if (matches > 1)
+ multi_pack++;
+ }
+
+ printf("%u packs, %u entries, %u%% unique\n",
+ pack_cnt, entry_cnt,
+ (entry_cnt - multi_pack) * 100 / entry_cnt);
+}
+
+static const char super_index_usage[] =
+"git-super-index";
+
+int cmd_super_index(int argc, char **argv, const char *prefix)
+{
+ select_packs();
+ if (!pack_cnt)
+ die("no packfiles to super-index");
+ if (pack_cnt == 1)
+ warning("only one packfile; a super-index is unnecessary");
+
+ while (compute_next_entry())
+ /* nothing */;
+ write_super_index();
+
+ if (0)
+ show_super_index();
+
+ return 0;
+}
diff --git a/builtin.h b/builtin.h
index d3f3a74..2a11187 100644
--- a/builtin.h
+++ b/builtin.h
@@ -71,6 +71,7 @@ extern int cmd_shortlog(int argc, const char **argv, const char *prefix);
extern int cmd_show(int argc, const char **argv, const char *prefix);
extern int cmd_show_branch(int argc, const char **argv, const char *prefix);
extern int cmd_stripspace(int argc, const char **argv, const char *prefix);
+extern int cmd_super_index(int argc, const char **argv, const char *prefix);
extern int cmd_symbolic_ref(int argc, const char **argv, const char *prefix);
extern int cmd_tar_tree(int argc, const char **argv, const char *prefix);
extern int cmd_unpack_objects(int argc, const char **argv, const char *prefix);
diff --git a/cache.h b/cache.h
index 0f4a05b..3c8a41d 100644
--- a/cache.h
+++ b/cache.h
@@ -490,7 +490,7 @@ extern unsigned char* use_pack(struct packed_git *, struct pack_window **, off_t
extern void unuse_pack(struct pack_window **);
extern struct packed_git *add_packed_git(const char *, int, int);
extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t);
-extern off_t find_pack_entry_one(const unsigned char *, struct packed_git *);
+extern off_t find_pack_entry_one(const unsigned char *, struct packed_git *, unsigned);
extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);
extern unsigned long unpack_object_header_gently(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);
extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);
diff --git a/git.c b/git.c
index 29b55a1..c323479 100644
--- a/git.c
+++ b/git.c
@@ -284,6 +284,7 @@ static void handle_internal_command(int argc, const char **argv, char **envp)
{ "show-branch", cmd_show_branch, RUN_SETUP },
{ "show", cmd_show, RUN_SETUP | USE_PAGER },
{ "stripspace", cmd_stripspace },
+ { "super-index", cmd_super_index, RUN_SETUP },
{ "symbolic-ref", cmd_symbolic_ref, RUN_SETUP },
{ "tar-tree", cmd_tar_tree },
{ "unpack-objects", cmd_unpack_objects, RUN_SETUP },
diff --git a/pack-check.c b/pack-check.c
index 7475348..ec61592 100644
--- a/pack-check.c
+++ b/pack-check.c
@@ -51,7 +51,7 @@ static int verify_packfile(struct packed_git *p,
sha1 = nth_packed_object_sha1(p, i);
if (!sha1)
die("internal error pack-check nth-packed-object");
- offset = find_pack_entry_one(sha1, p);
+ offset = find_pack_entry_one(sha1, p, 0);
if (!offset)
die("internal error pack-check find-pack-entry-one");
data = unpack_entry(p, offset, &type, &size);
@@ -94,7 +94,7 @@ static void show_pack_info(struct packed_git *p)
sha1 = nth_packed_object_sha1(p, i);
if (!sha1)
die("internal error pack-check nth-packed-object");
- offset = find_pack_entry_one(sha1, p);
+ offset = find_pack_entry_one(sha1, p, 0);
if (!offset)
die("internal error pack-check find-pack-entry-one");
diff --git a/pack.h b/pack.h
index d667fb8..b660833 100644
--- a/pack.h
+++ b/pack.h
@@ -43,6 +43,23 @@ struct pack_idx_header {
};
+/*
+ * Super-pack index header
+ *
+ * This file points to multiple packfiles by name and provides a
+ * hint to tell us if it is likely that a given packfile contains
+ * a particular object. This file is not an index file replacement,
+ * it is only a performance optimization hint.
+ */
+#define PACK_SDX_SIGNATURE 0x50534458 /* "PSDX" */
+struct pack_sdx_header {
+ uint32_t sdx_signature;
+ uint32_t sdx_version;
+ uint16_t sdx_packs;
+ uint8_t sdx_prefix;
+ uint8_t _unused_padding;
+};
+
extern int verify_pack(struct packed_git *, int);
extern void fixup_pack_header_footer(int, unsigned char *, const char *, uint32_t);
diff --git a/sha1_file.c b/sha1_file.c
index a3637d7..d296b32 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -405,6 +405,15 @@ static char *find_sha1_file(const unsigned char *sha1, struct stat *st)
return NULL;
}
+struct super_index {
+ struct pack_sdx_header *sdx_header;
+ void *sdx_data;
+ unsigned char *sdx_pack_names;
+ struct packed_git **packs;
+ unsigned pack_cnt;
+};
+
+static struct super_index *super_idx;
static unsigned int pack_used_ctr;
static unsigned int pack_mmap_calls;
static unsigned int peak_pack_open_windows;
@@ -831,6 +840,64 @@ void install_packed_git(struct packed_git *pack)
packed_git = pack;
}
+static int open_super_index(const char *path)
+{
+ int fd = open(path, O_RDONLY);
+ struct stat st;
+ size_t sdx_size;
+ void *sdx_map;
+ struct pack_sdx_header *hdr;
+ unsigned char *sdx_data;
+ unsigned pack_cnt;
+
+ if (fd < 0)
+ return -1;
+ if (fstat(fd, &st)) {
+ close(fd);
+ return -1;
+ }
+
+ sdx_size = xsize_t(st.st_size);
+ if (sdx_size < 4 * 256 + 20) {
+ close(fd);
+ return error("super-index %s is too small", path);
+ }
+ sdx_map = mmap(NULL, sdx_size, PROT_READ, MAP_PRIVATE, fd, 0);
+ if (sdx_map == MAP_FAILED) {
+ release_pack_memory(sdx_size, fd);
+ sdx_map = mmap(NULL, sdx_size, PROT_READ, MAP_PRIVATE, fd, 0);
+ if (sdx_map == MAP_FAILED) {
+ close(fd);
+ return -1;
+ }
+ }
+ close(fd);
+
+ hdr = sdx_map;
+ if (hdr->sdx_signature != htonl(PACK_SDX_SIGNATURE)) {
+ munmap(sdx_map, sdx_size);
+ return error("super-index %s has incorrect signature", path);
+ }
+ if (hdr->sdx_version != htonl(1)) {
+ munmap(sdx_map, sdx_size);
+ return error("super-index file %s is version %u"
+ " and is not supported by this binary"
+ " (try upgrading GIT to a newer version)",
+ path, ntohl(hdr->sdx_version));
+ }
+
+ pack_cnt = ntohs(hdr->sdx_packs);
+ sdx_data = (unsigned char*)(hdr + 1);
+
+ super_idx = xmalloc(sizeof(*super_idx));
+ super_idx->sdx_header = hdr;
+ super_idx->sdx_pack_names = sdx_data;
+ super_idx->sdx_data = sdx_data + pack_cnt * 20;
+ super_idx->pack_cnt = pack_cnt;
+ super_idx->packs = xcalloc(pack_cnt, sizeof(*super_idx->packs));
+ return 0;
+}
+
static void prepare_packed_git_one(char *objdir, int local)
{
char path[PATH_MAX];
@@ -852,6 +919,12 @@ static void prepare_packed_git_one(char *objdir, int local)
int namelen = strlen(de->d_name);
struct packed_git *p;
+ if (!super_idx && !strcmp(de->d_name, "super-index.sdx")) {
+ strcpy(path + len, de->d_name);
+ open_super_index(path);
+ continue;
+ }
+
if (!has_extension(de->d_name, ".idx"))
continue;
@@ -1260,7 +1333,7 @@ static off_t get_delta_base(struct packed_git *p,
*curpos += used;
} else if (type == OBJ_REF_DELTA) {
/* The base entry _must_ be in the same pack */
- base_offset = find_pack_entry_one(base_info, p);
+ base_offset = find_pack_entry_one(base_info, p, 0);
if (!base_offset)
die("failed to find delta-pack base object %s",
sha1_to_hex(base_info));
@@ -1362,7 +1435,7 @@ const char *packed_object_info_detail(struct packed_git *p,
next_sha1 = use_pack(p, &w_curs, curpos, NULL);
if (*delta_chain_length == 0)
hashcpy(base_sha1, next_sha1);
- obj_offset = find_pack_entry_one(next_sha1, p);
+ obj_offset = find_pack_entry_one(next_sha1, p, 0);
break;
}
(*delta_chain_length)++;
@@ -1633,11 +1706,12 @@ static off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)
}
off_t find_pack_entry_one(const unsigned char *sha1,
- struct packed_git *p)
+ struct packed_git *p,
+ unsigned shift)
{
const uint32_t *level1_ofs = p->index_data;
const unsigned char *index = p->index_data;
- unsigned hi, lo;
+ unsigned hi, lo, mi;
if (!index) {
if (open_pack_index(p))
@@ -1652,9 +1726,12 @@ off_t find_pack_entry_one(const unsigned char *sha1,
index += 4 * 256;
hi = ntohl(level1_ofs[*sha1]);
lo = ((*sha1 == 0x0) ? 0 : ntohl(level1_ofs[*sha1 - 1]));
+ if (shift)
+ mi = lo + (hi - lo) * (shift - 1) / 65535;
+ else
+ mi = (lo + hi) / 2;
do {
- unsigned mi = (lo + hi) / 2;
unsigned x = (p->index_version > 1) ? (mi * 20) : (mi * 24 + 4);
int cmp = hashcmp(index + x, sha1);
if (!cmp)
@@ -1663,6 +1740,7 @@ off_t find_pack_entry_one(const unsigned char *sha1,
hi = mi;
else
lo = mi+1;
+ mi = (lo + hi) / 2;
} while (lo < hi);
return 0;
}
@@ -1685,42 +1763,100 @@ static int matches_pack_name(struct packed_git *p, const char *ig)
return 1;
}
-static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e, const char **ignore_packed)
+static int find_in_pack(struct packed_git *p, unsigned shift, const unsigned char *sha1, struct pack_entry *e, const char **ignore_packed)
{
- struct packed_git *p;
off_t offset;
- prepare_packed_git();
+ if (ignore_packed) {
+ const char **ig;
+ for (ig = ignore_packed; *ig; ig++)
+ if (!matches_pack_name(p, *ig))
+ break;
+ if (*ig)
+ return 0;
+ }
- for (p = packed_git; p; p = p->next) {
- if (ignore_packed) {
- const char **ig;
- for (ig = ignore_packed; *ig; ig++)
- if (!matches_pack_name(p, *ig))
- break;
- if (*ig)
- continue;
- }
- offset = find_pack_entry_one(sha1, p);
- if (offset) {
- /*
- * We are about to tell the caller where they can
- * locate the requested object. We better make
- * sure the packfile is still here and can be
- * accessed before supplying that answer, as
- * it may have been deleted since the index
- * was loaded!
- */
- if (p->pack_fd == -1 && open_packed_git(p)) {
- error("packfile %s cannot be accessed", p->pack_name);
- continue;
+ offset = find_pack_entry_one(sha1, p, shift);
+ if (!offset)
+ return 0;
+
+ /*
+ * We are about to tell the caller where they can
+ * locate the requested object. We better make
+ * sure the packfile is still here and can be
+ * accessed before supplying that answer, as
+ * it may have been deleted since the index
+ * was loaded!
+ */
+ if (p->pack_fd == -1 && open_packed_git(p)) {
+ error("packfile %s cannot be accessed", p->pack_name);
+ return 0;
+ }
+ e->offset = offset;
+ e->p = p;
+ hashcpy(e->sha1, sha1);
+ return 1;
+}
+
+static int find_superindex_entry(const unsigned char *sha1, struct pack_entry *e, const char **ignore_packed)
+{
+ const uint32_t *level1_ofs = super_idx->sdx_data;
+ const unsigned char *index = super_idx->sdx_data;
+ const unsigned pfx_len = super_idx->sdx_header->sdx_prefix;
+ const unsigned packs = super_idx->pack_cnt;
+ unsigned hi, lo;
+
+ index += 4 * 256;
+ hi = ntohl(level1_ofs[*sha1]);
+ lo = ((*sha1 == 0x0) ? 0 : ntohl(level1_ofs[*sha1 - 1]));
+
+ do {
+ unsigned mi = (lo + hi) / 2;
+ unsigned x = mi * (pfx_len + packs);
+ int cmp = memcmp(index + x, sha1 + 1, pfx_len);
+ if (!cmp) {
+ index += x + pfx_len;
+ for (x = 0; x < packs; x++, index += 2) {
+ unsigned shift = (*index << 8) | index[1];
+ struct packed_git *p;
+ if (!shift)
+ continue;
+ p = super_idx->packs[x];
+ if (!p) {
+ for (p = packed_git; p; p = p->next) {
+ if (!hashcmp(p->sha1,
+ super_idx->sdx_pack_names + x * 20))
+ break;
+ }
+ if (!p)
+ p = MAP_FAILED;
+ super_idx->packs[x] = p;
+ }
+ if (p == MAP_FAILED)
+ continue;
+ if (find_in_pack(p, shift, sha1, e, ignore_packed))
+ return 1;
}
- e->offset = offset;
- e->p = p;
- hashcpy(e->sha1, sha1);
- return 1;
+ return 0;
}
- }
+ if (cmp > 0)
+ hi = mi;
+ else
+ lo = mi + 1;
+ } while (lo < hi);
+ return 0;
+}
+
+static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e, const char **ignore_packed)
+{
+ struct packed_git *p;
+
+ prepare_packed_git();
+ if (super_idx && find_superindex_entry(sha1, e, ignore_packed))
+ return 1;
+ for (p = packed_git; p; p = p->next)
+ if (find_in_pack(p, 0, sha1, e, ignore_packed))
+ return 1;
return 0;
}
@@ -1730,7 +1866,7 @@ struct packed_git *find_sha1_pack(const unsigned char *sha1,
struct packed_git *p;
for (p = packs; p; p = p->next) {
- if (find_pack_entry_one(sha1, p))
+ if (find_pack_entry_one(sha1, p, 0))
return p;
}
return NULL;
--
1.5.2.838.g8a923
^ permalink raw reply related
* Re: [PATCH] Add basic test-script for git-submodule
From: Junio C Hamano @ 2007-05-29 7:10 UTC (permalink / raw)
To: Lars Hjemli; +Cc: Johannes Schindelin, git
In-Reply-To: <11802980992216-git-send-email-hjemli@gmail.com>
Lars Hjemli <hjemli@gmail.com> writes:
> +# create a submodule repository
> +mkdir lib && cd lib
> +git-init >/dev/null
> +echo a >a && git-add a && git-commit -q -m "submodule commit 1"
> +git-tag -a -m "rev-1" rev-1
> +rev1=$(git-rev-parse HEAD)
> +cd ..
> +
> +# add submodule and other files to super repo
> +echo a >a && echo z >z
> +git-add a lib z && git-commit -q -m "super commit 1"
> +
> +# move submodule to another location, register repo url in .gitmodules
> +mv lib .subrepo
> +GIT_CONFIG=.gitmodules git-config module.lib.url ./.subrepo
We typically try to catch malfunction even while setting up the test case.
> +test_expect_success 'status is "missing"' \
> + 'git-submodule status | grep "^-$rev1"'
> +
> +# make sure 'init' will not overwrite a regular file
> +touch lib
> +test_expect_failure 'init fails when path is used by a file' \
> + 'git-submodule init'
I am guilty for introducing "expect-failure", but it is usually
a mistake to use it unless you are testing something very trivial.
For example, for this case, you would want to make sure the
command "git-submodule init" exits with non-zero status, but
also you would want to make sure that it does not disturb the
existing file "lib". You might also want to see if the command
gives the right error message (or status code) as well, although
typically the wording of error message is subject to change, so
we tend to try not to be too strict about it.
In any case, I would probably write this part of the code like
this:
test_expect_success 'init fails when path is used by a file' '
echo hello >lib &&
if git-submodule init
then
echo "Oops, should have failed"
false
elif test -f lib && test "$(cat lib)" = hello
then
: happy
else
echo "Oops, failed but lib file was molested"
false
fi
'
Later when somebody tries to improve git-submodule and botches,
he can run the test with "-i -v" and observe which "Oops" comes
out to see why the test failed -- it would give him a clue on
how he broke it.
^ permalink raw reply
* Re: [PATCH 00/15] git-note: A mechanisim for providing free-form after-the-fact annotations on commits
From: Johan Herland @ 2007-05-29 7:06 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Linus Torvalds, git
In-Reply-To: <7vwsysbrtg.fsf@assigned-by-dhcp.cox.net>
On Monday 28 May 2007, Junio C Hamano wrote:
> Johan Herland <johan@herland.net> writes:
> > On Monday 28 May 2007, Linus Torvalds wrote:
> >> On Mon, 28 May 2007, Johan Herland wrote:
> >> > I still don't see what makes note objects inherently more expensive than
> >> > commit objects. Except for the refs, of course, but we're getting rid
> >> > of those (at least replacing them with a more efficient reverse mapping).
> >>
> >> It's exactly the refs that I worry about.
> >>
> >> Anything that needs to read in all notes at startup is going to be _slow_.
> >>
> >> In contrast, commits we read when (and only when) we need them.
> >
> > Ok. But the reverse mapping will help with this, won't it?
> > We'll look up the interesting commits and find their associated
> > note objects directly.
>
> The issue Linus brought up worries me, too.
>
> The "efficient reverse mapping" is still handwaving at this
> stage. What it needs to do is an equivalent to your
> implementation with "refs/notes/<a dir per commit>/<note>". The
> "efficient" one might do a flat file that says "notee note" per
> line sorted by notee, or it might use BDB or sqlite, but the
> amount of the data and complexity of the look-up is really the
> same. A handful notes per each commit in the history (I think
> Linus's "Acked-by after the fact" example a very sensible thing
> to want from this subsystem).
>
> I am not saying that it is impossible to make the set-up cost
> for the "efficient lookup" almost zero, and to make it lazy and
> on-demand. The concern above just adds one design constraints
> to that "efficient reverse mapping" code yet to come.
Ok, here's what I'm thinking so far on that reverse mapping:
1. Keep a file, ".git/reverse_tagmap_sorted" with one entry of the form
"pointee pointer" per line. The file is sorted on "pointee", so we can
easily do the radix-256-fan-out-followed-by-binary-search trick that
Linus mentioned in another thread. This should hopefully make lookup
fairly cheap. BTW, if there is a similar "pointee pointer"-type format
already being used in git, I'd be happy to use that instead. I looked
at the "peeled" format being used by packed-refs, but using that
directly doesn't sound like a good idea, since the refname causes the
entries to be of variable length, and the refnames are not interesting
to me at all.
2. Keep another file, ".git/reverse_tagmap_unsorted" in front of (1).
This file has exactly the same format, minus the sorting. It exists just
to make insertion cheap. Once this file reaches a certain size (i.e.
when trawling it on lookup becomes slightly painful), we shuffle the
entries into the sorted file (this would happen automatically on
insertion of an entry, and should _not_ have to be triggered by 'git-gc'
etc.).
Of course, if we think insertion directly into (1) will never be too
expensive, we can drop (2) altogether.
I don't know enough about packing to have a good idea on how to pack
these reverse tagmaps, but Shawn's thoughts about keeping associated
tags/notes and objects close together makes a lot of sense. I'm just
not sure yet where these reverse tagmaps fit into the whole picture.
Currently, AFAICS, the packed-refs file is never propagated into the
packs, but stays separate for the lifetime of the repo, but then it
seems we're designing these reverse tagmaps for managing a handful of
notes per commit, i.e. to hold a couple of orders of magnitude more
entries than the packed-refs file.
Maybe each pack should keep the reverse tagmap for all the object->note
relationships internal to that pack? Everything else (unpacked notes,
and object->note relationships spanning packs) would be kept in (1).
Of course, when repacking, we'd try to keep objects and their notes
together as much as possible, to maximize the in-pack reverse tagmap,
and minimize the number of entries left behind in (1).
Have fun!
...Johan
--
Johan Herland, <johan@herland.net>
www.herland.net
^ permalink raw reply
* Re: [PATCH 00/15] git-note: A mechanisim for providing free-form after-the-fact annotations on commits
From: Jakub Narebski @ 2007-05-29 7:04 UTC (permalink / raw)
To: git
In-Reply-To: <20070529032221.GE7044@spearce.org>
Shawn O. Pearce wrote:
> Which is why we:
>
> a) allow these things to migrate into packed-refs, because
> getting into there is a hellva lot cheaper than getting into
> a packfile;
>
> b) move them into a packfile when we repack loose objects,
> because then we have really good access.
>
> c) take them out of packed-refs once they are into a packfile,
> and get them out of the loose refs/notes directory as early as
> possible.
Three-tier layout? Loose note refs -> packed refs -> packfile?
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
^ permalink raw reply
* Re: [PATCH] move template_dir logic to configure script.
From: Junio C Hamano @ 2007-05-29 6:46 UTC (permalink / raw)
To: hanwen; +Cc: git
In-Reply-To: <4658BDB0.9040501@xs4all.nl>
Han-Wen Nienhuys <hanwen@xs4all.nl> writes:
> ---
Could I have at least three lines above the --- marker, please?
(1) Defend _why_ this is a good change --- I would agree that
this would make it more flexible, and I suspect that your
main thrust with this is on some platforms (perhaps
Windows?) you would want to have template_dir that is
somewhere away from datarootdir.
(2) an empty line between (1) and (3)
(3) Signed-off-by: line.
> config.mak.in | 2 +-
> configure.ac | 6 ++++++
> 2 files changed, 7 insertions(+), 1 deletions(-)
^ permalink raw reply
* Re: What's new in git-gui.git
From: Shawn O. Pearce @ 2007-05-29 4:11 UTC (permalink / raw)
To: git
In-Reply-To: <20070528224949.GC7044@spearce.org>
"Shawn O. Pearce" <spearce@spearce.org> wrote:
> --- Roadmap ---
>
> Looking ahead, I'd like to see some improvement in the following
> areas. It would be nice if some Tcl/Tk-minded individuals were
> willing to dive and help. ;-)
I also want to get file change notification integrated into git-gui.
Basic idea is, if you edit a file in your editor of choice, git-gui
should notice it right away, without waiting for you to press the
damn Rescan button.
On Windows the only thing that actually works there (aside from
drawing lots and lots of blue pixels) is a fancy little file
monitoring API. That actually works quite well. Nothing else on
that OS works of course, but it will tell you when a file changes.
NT 4.0 and later only... ;-)
Apparently the Linux folks have this new fangled inotify thing
that is also good at telling applications about changed files.
It seems the hg folks are using it to speed up "hg status" by
having a background daemon keep track of which files are dirty,
and which aren't.
So I'm likely to want to try to implement something like this for
core Git and for git-gui, but my primary development system is Mac
OS X, where the BSD gods have only given us kqueue() and all its
warts... (it is basically useless for this task of monitoring a
"huge" tree like git.git, Mozilla.git need not apply).
--
Shawn.
^ permalink raw reply
* Re: [PATCH] Don't ignore write failure from git-diff, git-log, etc.
From: Linus Torvalds @ 2007-05-29 3:47 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Jim Meyering, git
In-Reply-To: <7v1wh0bpv2.fsf@assigned-by-dhcp.cox.net>
On Mon, 28 May 2007, Junio C Hamano wrote:
>
> I think something like this instead might be more palatable.
>
> diff --git a/write_or_die.c b/write_or_die.c
> index 5c4bc85..fadfcaa 100644
> --- a/write_or_die.c
> +++ b/write_or_die.c
> @@ -41,8 +41,8 @@ int write_in_full(int fd, const void *buf, size_t count)
> void write_or_die(int fd, const void *buf, size_t count)
> {
> if (write_in_full(fd, buf, count) < 0) {
> - if (errno == EPIPE)
> - exit(0);
> - die("write error (%s)", strerror(errno));
> + if (errno != EPIPE)
> + die("write error (%s)", strerror(errno));
> + exit(1);
> }
> }
It's certainly less annoying, but I don't actually think it's *correct*.
I think the current behaviour is simply _superior_.
Think about what happens whenever somebody does
git cmd | head
and then you want to see whether the end result was successful or not.
What would the error code from the "git cmd" thing mean?
I claim that exiting with SUCCESS is actually the rigt thing to do for the
git command in question. It did what it was asked for.
And more importantly, considering that a pipe is of indeterminate size,
what happens if it actuall yonly had 15 lines to be printed out. Guess
what? If it writes them fast enough, they'll go into the pipe, and "head"
will exit only after the write, and we'll never even *know* that the last
five lines were ignored, ie there won't be a EPIPE at all.
In other words, even Jim's example of
git log | sync
will actually *succeed* even with Jim's patch, if the output fit in the
kernel pipe buffer, and the "git log" command ran quickly enough that
"sync" took longer (which is not at all unlikely).
So EPIPE really _is_ special: because when you write to a pipe, there's no
guarantee that you'll get it at all, so whenever you get an EPIPE you
should ask yourself:
- what would I have done if the data had fit in the 64kB kernel buffer?
- should I really return a different error message or complain just
because I just happened to notice that the reader went away _this_
time, even if I might not notice it next time?
In other words, the "exit(0)" is actually _more_ consistent than
"exit(1)", because exiting with an error message or with an error return
is going to depend on luck and timing.
For example, I just did a
strace git show | sync
on my kernel archive, and strace shows me that I had:
...
write(1, "commit c420bc9f09a0926b708c3edb2"..., 736) = 736
exit_group(0)
the first three times I tried it, and then
write(1, "commit c420bc9f09a0926b708c3edb2"..., 736) = -1 EPIPE (Broken pipe)
--- SIGPIPE (Broken pipe) @ 0 (0) ---
+++ killed by SIGPIPE +++
the fourth time.
What should we learn from this?
- a shell that talks about SIGPIPE is going to be universally *hated*,
because it's really a total crap-shoot. Even with something like
"sync", that doesn't read a single byte from the input, most of the
time I didn't actually get EPIPE/SIGPIPE at all!
- by exactly the same token, considering "EPIPE" as anything else than a
"success" is always just going to lead you to random behaviour.
So what _should_ you do for EPIPE?
Here's what EPIPE _really_ means:
- you might as well consider the write a success, but the reader isn't
actually interested, so rather than go on, you might as well stop
early.
Notice how I very carefull avoided the word "error" anywhere. Because it's
really not an error. The reader already got everything it wanted. So EPIPE
should generally be seen as an "early success" rather than as a "failure".
Linus
^ permalink raw reply
* Re: [PATCH 1/3] Lazily open pack index files on demand
From: Linus Torvalds @ 2007-05-29 3:26 UTC (permalink / raw)
To: linux; +Cc: git
In-Reply-To: <20070529000949.28007.qmail@science.horizon.com>
On Mon, 28 May 2007, linux@horizon.com wrote:
>
> Even losing a constant factor of 2, it still seems like it might offer a
> factor-of-2 speedup for large repositories.
Well, not so much according to the numbers I had.
Yes, SHA-1's are very nicely distributed on a larger scale, but on a
_small_ scale they aren't.
So you end up being able to get very good initial guesses for the first
few iterations, but once you get "close enough", you can't do anything at
all, and then you're actually worse off.
Also, please do realize that the binary search is actually a *smart*
binary search, which does a radix-256 fan-out at the beginning, getting
rid of the first 8 levels of searching for free.
The Newton-Raphson thing can do that too (and in my trial patch, did), but
it means that you actually get rid of just the initial guess (the one that
worked the best!) and you still have the problem that once you get close
enough, the distribution at a local level is not at all nice and linear.
So what I did with my patch (you can find it in the archives - search for
Newton-Raphson, I'd say about 3 months ago) was to do three cycles of
approximating, and then a linear search. The linear search has good cache
behaviour, so it's not as horrid as it might sound, but for some numbers I
did, my approximate thing would hit on the exact first entry about half
the time, but would have to walk up to ~20 entries occasionally.
Which meant that the binary search (with the initial radix-256 fan-out)
actually mostly outperformed the Newton-Raphson thing.
Also, object lookup is certainly noticeable, but the biggest cost of it is
the cache misses, and even then it's not really _that_ noticeable. And
it's really neither slow nor stupid as it is.
So I'd love to see somebody try to do a _proper_ apprixmation of Newton-
Raphson, not the quick hack I did. But I think factor-of-two is actually
optimistic, even for pretty large repos. And it would have to be no worse
than what we have now for average-sized ones.
(And object lookup time is generally not the dominant cost, so while it's
noticeable, cutting it down by even a whole 50% isn't going to speed up
any normal git operations by more than a couple of percent.. Generating
diffs in particular is a much more costly op for things like "git blame")
Linus
^ permalink raw reply
* Re: [PATCH 00/15] git-note: A mechanisim for providing free-form after-the-fact annotations on commits
From: Shawn O. Pearce @ 2007-05-29 3:22 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio C Hamano, Johan Herland, git
In-Reply-To: <alpine.LFD.0.98.0705282008400.26602@woody.linux-foundation.org>
Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
> On Mon, 28 May 2007, Shawn O. Pearce wrote:
> >
> > What if we use a "slow" storage by "refs/notes/$objname/$notename",
>
> This _really_ won't scale. Even if the notes don't exist, just doing the
> lookup (which will fail for most commits) will be horribly slow, and will
> populate the dentry cache with negative entries.
Yes.
I think you missed what I was trying to say. We *definately* do not
want to probe the OS and ask "do you have $objname1? $objname2?"
for exactly the reason you just stated. (Though you being some
sort of kernel guru means you know a hellva lot more about that
dentry cache thing than I do!)
What I meant was something more like:
DIR *d = opendir("refs/notes");
if (d) {
... notes can exist in both packfiles and "loose" ...
only_packed_notes = 0;
while (readdir(d)) {
... the entry name here is the name of an object ...
... stuff into a hash table, we can come back later ...
... for the subdirectory contents if we need it ...
}
closedir(d);
... also examine "packed-refs", in case any are there ...
} else {
only_packed_notes = 1;
... we *only* have notes in packfiles, if we have any at all...
}
Then looking up a note is just a probe into our in-memory hash
(if only_packed_notes is false) and a probe into the packfile(s) to
find the notes for the object. Not very expensive if the packfiles
have the reverse obj->tag mappings indexed within them.
> To get good filesystem performance, you have to
> - have reasonable hit-rates (and looking it up for each commit is _not_
> going to do that)
> - not have millions of objects.
Which is why we:
a) allow these things to migrate into packed-refs, because
getting into there is a hellva lot cheaper than getting into
a packfile;
b) move them into a packfile when we repack loose objects,
because then we have really good access.
c) take them out of packed-refs once they are into a packfile,
and get them out of the loose refs/notes directory as early as
possible.
--
Shawn.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox