* Re: [PATCH] Skip writing out sha1 files for objects in packed git.
From: Linus Torvalds @ 2005-06-28 15:45 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vekanyx33.fsf@assigned-by-dhcp.cox.net>
On Mon, 27 Jun 2005, Junio C Hamano wrote:
>
> LT> And if we have a pack-file in .git/objects/ that already has
> LT> the object, that may not be the _same_ pack-file that we're
> LT> expanding at all, so if that pack file already has the
> LT> object, then not writing it out is actually the right thing
> LT> to do.
>
> This I have to think about a bit.
The most trivial example is doing a "git pull" of a small pack-file
update.
We probably don't want to leave it around as a pack-file (we'll re-pack
everything at some later date, but we also don't want to expand the stuff
we already have in our _real_ pack-file).
Linus
^ permalink raw reply
* Re: Mercurial vs Updated git HOWTO for kernel hackers
From: Petr Baudis @ 2005-06-28 15:35 UTC (permalink / raw)
To: Andrew Thompson; +Cc: Jeff Garzik, mercurial, Git Mailing List, Linux Kernel
In-Reply-To: <42C16877.6000909@aktzero.com>
Dear diary, on Tue, Jun 28, 2005 at 05:10:47PM CEST, I got a letter
where Andrew Thompson <andrewkt@aktzero.com> told me that...
> Petr Baudis wrote:
> >>Mercurial's undo is taking a snapshot of all the changed file's repo file
> >>length
> >>at every commit or pull. It just truncate the file to original size and
> >>undo is done.
> >
> >"Trunactes"? That sounds very wrong... you mean replace with old
> >version? Anyway, what if the file has same length? It just doesn't make
> >much sense to me.
>
> I believe this works because the files stored in a binary format that
> appends new changesets onto the end. Thus, truncating the new stuff from
> the end effectively removes the commit.
Yes, I'm sorry - I missed the "repo" part and thought that was what it
was doing with the checked out files. ;-)
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
<Espy> be careful, some twit might quote you out of context..
^ permalink raw reply
* Re: Mercurial vs Updated git HOWTO for kernel hackers
From: Petr Baudis @ 2005-06-28 15:34 UTC (permalink / raw)
To: Kevin Smith, Git Mailing List, mercurial
In-Reply-To: <20050628151522.GO5992MdfPADPa@garage.linux.student.kuleuven.ac.be>
Dear diary, on Tue, Jun 28, 2005 at 05:15:22PM CEST, I got a letter
where Sven Verdoolaege <skimo@kotnet.org> told me that...
> On Tue, Jun 28, 2005 at 05:07:52PM +0200, Petr Baudis wrote:
> > Dear diary, on Fri, Jun 24, 2005 at 03:57:00PM CEST, I got a letter
> > where Kevin Smith <yarcs@qualitycode.com> told me that...
> > > - Can run on (native) MS Windows
> > > (necessary for me because I often work on cross-platform projects)
> >
> > I'd expect everything to work fine with Cygwin (or with only minor
> > problems easy to fix) or just any working bash + GNU coreutils
> > installation. Any issue with that?
>
> The code is full of Unixisms. You'd almost think that the authors
> of git have some kind of affinity towards Unix.
Ah. Well, I was speaking just of Cogito and didn't realize you might
want git too so that you can actually do anything with Cogito then. ;-)
> read-cache.c:376: warning: implicit declaration of function `mmap'
Well, I'm actually not very clueful about MS Windows environment, but
doesn't it provide any POSIX API? You might have more luck with that.
Or I guess it should work on Cygwin.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
<Espy> be careful, some twit might quote you out of context..
^ permalink raw reply
* Re: Mercurial vs Updated git HOWTO for kernel hackers
From: Sven Verdoolaege @ 2005-06-28 15:15 UTC (permalink / raw)
To: Petr Baudis; +Cc: Kevin Smith, Git Mailing List, mercurial
In-Reply-To: <20050628150752.GC1275@pasky.ji.cz>
On Tue, Jun 28, 2005 at 05:07:52PM +0200, Petr Baudis wrote:
> Dear diary, on Fri, Jun 24, 2005 at 03:57:00PM CEST, I got a letter
> where Kevin Smith <yarcs@qualitycode.com> told me that...
> > - Can run on (native) MS Windows
> > (necessary for me because I often work on cross-platform projects)
>
> I'd expect everything to work fine with Cygwin (or with only minor
> problems easy to fix) or just any working bash + GNU coreutils
> installation. Any issue with that?
The code is full of Unixisms. You'd almost think that the authors
of git have some kind of affinity towards Unix.
i386-mingw32-gcc -g -O2 -Wall '-DSHA1_HEADER=<openssl/sha.h>' -c -o read-cache.o read-cache.c
In file included from read-cache.c:6:
cache.h:14:22: sys/mman.h: No such file or directory
cache.h:16:24: netinet/in.h: No such file or directory
In file included from read-cache.c:6:
cache.h: In function `create_ce_mode':
cache.h:104: warning: implicit declaration of function `S_ISLNK'
cache.h:105: warning: implicit declaration of function `htonl'
cache.h:105: error: `S_IFLNK' undeclared (first use in this function)
cache.h:105: error: (Each undeclared identifier is reported only once
cache.h:105: error: for each function it appears in.)
[..]
read-cache.c:376: warning: implicit declaration of function `mmap'
read-cache.c:376: error: `PROT_READ' undeclared (first use in this function)
read-cache.c:376: error: `PROT_WRITE' undeclared (first use in this function)
read-cache.c:376: error: `MAP_PRIVATE' undeclared (first use in this function)
read-cache.c:376: warning: assignment makes pointer from integer without a cast
read-cache.c:399: warning: implicit declaration of function `munmap'
[..]
skimo
^ permalink raw reply
* Re: Mercurial vs Updated git HOWTO for kernel hackers
From: Andrew Thompson @ 2005-06-28 15:10 UTC (permalink / raw)
To: Petr Baudis; +Cc: Jeff Garzik, mercurial, Git Mailing List, Linux Kernel
In-Reply-To: <20050628150027.GB1275@pasky.ji.cz>
[-- Attachment #1: Type: text/plain, Size: 585 bytes --]
Petr Baudis wrote:
>>Mercurial's undo is taking a snapshot of all the changed file's repo file length
>>at every commit or pull. It just truncate the file to original size and undo
>>is done.
>
> "Trunactes"? That sounds very wrong... you mean replace with old
> version? Anyway, what if the file has same length? It just doesn't make
> much sense to me.
I believe this works because the files stored in a binary format that
appends new changesets onto the end. Thus, truncating the new stuff from
the end effectively removes the commit.
--
Andrew Thompson
http://aktzero.com/
[-- Attachment #2: andrewkt.vcf --]
[-- Type: text/x-vcard, Size: 153 bytes --]
begin:vcard
fn:Andrew Thompson
n:Thompson;Andrew
email;internet:andrewkt@aktzero.com
x-mozilla-html:FALSE
url:http://aktzero.com/
version:2.1
end:vcard
[-- Attachment #3: Type: text/plain, Size: 139 bytes --]
_______________________________________________
Mercurial mailing list
Mercurial@selenic.com
http://selenic.com/mailman/listinfo/mercurial
^ permalink raw reply
* Re: Mercurial vs Updated git HOWTO for kernel hackers
From: Petr Baudis @ 2005-06-28 15:07 UTC (permalink / raw)
To: Kevin Smith; +Cc: Git Mailing List, mercurial
In-Reply-To: <42BC112C.1040009@qualitycode.com>
Dear diary, on Fri, Jun 24, 2005 at 03:57:00PM CEST, I got a letter
where Kevin Smith <yarcs@qualitycode.com> told me that...
> - Can run on (native) MS Windows
> (necessary for me because I often work on cross-platform projects)
I'd expect everything to work fine with Cygwin (or with only minor
problems easy to fix) or just any working bash + GNU coreutils
installation. Any issue with that?
> - Python code can be more clear and expressive (IMHO)
>
> In the long run, I think the python code base will be easier to maintain
> and enhance. A rewrite of cogito in python or ruby would be cool.
I've planned to rewrite Cogito in Perl, but it turned out the shell
scripts really are far from the PITA they appeared to be - they work
just fine and feel quite comfortable from the maintenance standpoint as
well. It surprised me too but I don't plan to rewrite Cogito in a
different language soon.
> One advantage that cogito has is that git viewing/browsing tools can
> operate directly on cogito repos. But a psychological drawback is the
> ongoing confusion between git and cogito. Questions: Would a git-based
> tool that writes to the repo (such as StGIT) mess up a cogito repo? Can
> you switch a repo between git and cogito or back, at any time?
Cogito's only unusual requirement (well, expectation) is that HEAD is a
symlink to .git/refs/heads/master, and .git/refs/heads/master should
reflect your current head. I will try to ease up this restriction so
that things will mostly work even if you just have HEAD. I think that
most auxiliary commands (e.g. cg-log - you just have to love it) should
work on any sensible git tree (but I didn't test it - yet).
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
<Espy> be careful, some twit might quote you out of context..
^ permalink raw reply
* Re: Mercurial vs Updated git HOWTO for kernel hackers
From: Petr Baudis @ 2005-06-28 15:00 UTC (permalink / raw)
To: Christopher Li; +Cc: mercurial, Jeff Garzik, Linux Kernel, Git Mailing List
In-Reply-To: <20050624123819.GD9519@64m.dyndns.org>
Dear diary, on Fri, Jun 24, 2005 at 02:38:19PM CEST, I got a letter
where Christopher Li <hg@chrisli.org> told me that...
> On Fri, Jun 24, 2005 at 08:41:01AM +0200, Petr Baudis wrote:
> > > 5.1) undo the last commit or pull
> > >
> > > $ hg undo
> >
> > $ cg-admin-uncommit
> >
> > Note that you should never do this if you already pushed the changes
> > out, or someone might get them. (That holds for regular Git too.) See
> >
> > $ cg-help cg-admin-uncommit # (or cg-admin-uncommit --help)
> >
> > for details. (That's another Cogito's cool feature. Handy docs! ;-)
> >
>
> Does it still works if the last commit was interrupted or due to error for some
> reason?
If the last commit was interrupted, it didn't happen, so your tree stays
in the same state as before doing the commit, as well as the repository.
You can just try again. If you want to get rid of dirty stuff,
cg-cancel.
> Undo pull is pretty cool because you might pull a lot of commit
> in one blow. Get rid of commit one by one is going to be painful. Some times
> the object you pull has more than one chain of history it will be very nasty
> if you want to clean it up.
If it was a tree merge, cg-admin-uncommit will undo it. If it was
fast-forward merge, there is no direct way to uncommit it, but you can
find the first fast-forwarded commit and pass it as argument to
cg-admin-uncommit; it will then rewind all the commits up to (including)
the given commit.
> Mercurial's undo is taking a snapshot of all the changed file's repo file length
> at every commit or pull. It just truncate the file to original size and undo
> is done.
"Trunactes"? That sounds very wrong... you mean replace with old
version? Anyway, what if the file has same length? It just doesn't make
much sense to me.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
<Espy> be careful, some twit might quote you out of context..
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Petr Baudis @ 2005-06-28 14:52 UTC (permalink / raw)
To: Christopher Li; +Cc: Junio C Hamano, Linus Torvalds, git
In-Reply-To: <20050628110625.GC21533@64m.dyndns.org>
Dear diary, on Tue, Jun 28, 2005 at 01:06:25PM CEST, I got a letter
where Christopher Li <git@chrisli.org> told me that...
> On Tue, Jun 28, 2005 at 02:40:56AM -0700, Junio C Hamano wrote:
> > >>>>> "LT" == Linus Torvalds <torvalds@osdl.org> writes:
> > Still a good space reduction. Good job!
> >
> > I am now dreaming if we someday would enhance the mechanism with
> > append-only updates to the *.pack files with complete rewrite of
> > the *.idx files, and get rid of files under .git/objects totally.
>
> No offense my friend, this has been done. It's name is mercurial.
>
> > This would make things reasonably friendly to rsync. The kernel
> > pack has around 60M pack with 1.1M index, so everyday use would
> > involve incremental updates to the pack [*1*] and full download
> > of the index file.
>
> It still have other open issue. Now it would be harder to not sync
> all the heads. If I just want the clean Linus-2.6 tree, I have to
> dig it out from the pack file which mixing with other heads.
>
> You could host different projects with it's own pack file. That
> will lost the space saving on co-hosting projects.
>
> So I am not convince rsync is the way to go in long run. You need
> to have your own network syncing method.
I think the git-*-pull tools are actually just fine. You will only need
to have some server-side CGI gadget to frontend the file, but we need
that anyway to make the pull reasonably effective.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
<Espy> be careful, some twit might quote you out of context..
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Jan Harkes @ 2005-06-28 14:46 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Linus Torvalds, git
In-Reply-To: <7vekamvmxj.fsf@assigned-by-dhcp.cox.net>
On Tue, Jun 28, 2005 at 02:40:56AM -0700, Junio C Hamano wrote:
> I am now dreaming if we someday would enhance the mechanism with
> append-only updates to the *.pack files with complete rewrite of
> the *.idx files, and get rid of files under .git/objects totally.
Stop dreaming, please.
The current separate objects setup might not be space efficient, but it
has many other advantages.
- Objects are only written only once, and from then on are only read.
This works well on filesystems that provide session semantics, as
opposed to unix semantics. And the resulting objects are perfectly
cacheable since they are only invalidated if someone ever decides to
pack the repository.
- The hierarchy and the way the objects directories are updated works
very well in combination with AFS style directory acls. What surprised
me was that subdirectories in refs/heads work perfectly with all the
core git tools, branchnames simply become 'user/branch'.
- Objects that differ in content have different naming, as a result
multiple developers can safely commit into a shared repository without
requiring locks. This is also why it is safe to pull from another
repository without clobbering your own history. Imagine if you
appended some local changes to a packed archive and the next rsync
wipes your local commits.
I've been trying to keep an up to date document on how (and why) I use
git on Coda. It started pretty much the identical to jgarzik's HOWTO.
But it ended up a lot more complicated, to a point where I needed my own
scripts for just about every action. Until I discovered that the
alternate objects pool would work well in my environment.
http://www.coda.cs.cmu.edu/git.html
Jan
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Christopher Li @ 2005-06-28 11:06 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Linus Torvalds, git
In-Reply-To: <7vekamvmxj.fsf@assigned-by-dhcp.cox.net>
On Tue, Jun 28, 2005 at 02:40:56AM -0700, Junio C Hamano wrote:
> >>>>> "LT" == Linus Torvalds <torvalds@osdl.org> writes:
> Still a good space reduction. Good job!
>
> I am now dreaming if we someday would enhance the mechanism with
> append-only updates to the *.pack files with complete rewrite of
> the *.idx files, and get rid of files under .git/objects totally.
No offense my friend, this has been done. It's name is mercurial.
> This would make things reasonably friendly to rsync. The kernel
> pack has around 60M pack with 1.1M index, so everyday use would
> involve incremental updates to the pack [*1*] and full download
> of the index file.
It still have other open issue. Now it would be harder to not sync
all the heads. If I just want the clean Linus-2.6 tree, I have to
dig it out from the pack file which mixing with other heads.
You could host different projects with it's own pack file. That
will lost the space saving on co-hosting projects.
So I am not convince rsync is the way to go in long run. You need
to have your own network syncing method.
>
> [Footnote]
>
> *1* Presumably many objects are deltified against older objects
> which is suboptimal. Most likely the newer objects are accessed
> far more often and they are what we would want to keep in full
> not as delta. So even with this scheme we would want to have
> weekly repacking. Interestingly enough, pack-objects gets the
> objects via usual read_sha1_file() interface so it can produce a
> new pack from an existing pack.
It sounds like you are suggesting backward delta. Keeping the
latest node in full and using delta to access the old one. It should
work but it will lose the append only property.
Chris
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Christopher Li @ 2005-06-28 10:38 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Git Mailing List
In-Reply-To: <Pine.LNX.4.58.0506272016420.19755@ppc970.osdl.org>
That is all nice improvement to address the space usage issue.
Should people just run repacking once a while or is it automaticly
add new object to the pack file?
Chris
On Mon, Jun 27, 2005 at 08:30:22PM -0700, Linus Torvalds wrote:
>
> Deltas do exist inside pack-files, yes. They just don't exist as
> independent objects any more, so you can never get into the situation that
> you find a delta but you don't find the delta it points to.
>
> Because in the pack-files, there are only deltas _within_ a pack-file. You
> can't have a delta that points to outside the pack.
>
> This means that pack-files with few objects will inevitably be larger than
> they could otherwise be (ie you can never have a pack file that _only_
> contains deltas to the outside world), but it's just incredibly reassuring
> to me that a pack-file is always self-sufficient.
>
> So when/if we start using pack-files for doing "git pull" etc, the
> pack-file won't actually help pack things for small updates: small updates
> will probably contain the whole changed file, unless the update has
> several changes to the same file (which is not unusual, of course), in
> which case it will only contain one version and then deltas from that.
>
> But the savings get increasingly bigger the more history we have. That's
> also why the packed git archive is about 1/14th of the size of the fully
> unpacked disk usage of the git project, but a packed kernel archive "only"
> achieves a packing rate of 1/5th of the fully unpacked kernel archive. The
> git archive is all history, while the kernel archive just "appears", and
> 2/3 of the files have only one single version and thus don't delta-
> compress at all.
>
> (Another reason is probably that the kernel has bigger files, which means
> that it thus has relatively less loss in filesystem block padding).
>
> But not having any outside deltas not only makes me feel safer, it also
> means that you can fully validate a pack archive consistency without even
> knowing what project it is from - you can check the SHA1 results of every
> file in the pack against the index of the pack, and check that the SHA1's
> of the pack files themselves are valid. Again, this is just a data
> _consistency_ check, of course - it means that you can validate that it
> downloaded fine, and that you don't have disk corruption, but it doesn't
> mean that the data isn't evil and nasty and buggy ;)
>
> Linus
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: Stacked GIT 0.1 (a.k.a. quilt for git)
From: Catalin Marinas @ 2005-06-28 10:03 UTC (permalink / raw)
To: Paul Jackson; +Cc: git
In-Reply-To: <20050624045627.6e9cbaff.pj@sgi.com>
Paul Jackson <pj@sgi.com> wrote:
>> there is no way to modify the log history.
>
> Aha. If that means what I think it does, then I suspect I will remain
> with quilt. The per-patch comment is often about the last thing that
> I put in the patch.
OK, you convinced me :-). I will upload a new StGIT version tonight. I
removed the 'commit' command completely. To add changes into the
repository, you would use the 'refresh' command. There is only one git
commit object for each patch and they can be seen with tools like
cg-log on top of the stack base (i.e. the HEAD of the pulled tree).
There is no per-patch history anymore. The patch diff and information
(description, author name etc.) are indefinitely changeable via the
'refresh' command but only one commit with the latest information is
seen in the tree log for each patch.
To pull new changes from the master repository:
stg pop -a
cg-udpate (or whatever people use to pull and merge)
stg push -a
The 'push' command re-bases all the commits corresponding to the StGIT
patches.
Does this make it closer to quilt in functionality?
--
Catalin
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Junio C Hamano @ 2005-06-28 9:40 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <Pine.LNX.4.58.0506272016420.19755@ppc970.osdl.org>
>>>>> "LT" == Linus Torvalds <torvalds@osdl.org> writes:
LT> But the savings get increasingly bigger the more history we have. That's
LT> also why the packed git archive is about 1/14th of the size of the fully
LT> unpacked disk usage of the git project,...
GIT archive may be an odd-ball because the project itself is so
small, but a fair comparison should include the disk usage of
256 fan-out directories. Counting them, empty .git/objects/
with the 1.4MB packed archive and 90KB index file ends up being
somewhere around 2.4MB on my machine, compared with 17MB for the
traditional one.
Still a good space reduction. Good job!
I am now dreaming if we someday would enhance the mechanism with
append-only updates to the *.pack files with complete rewrite of
the *.idx files, and get rid of files under .git/objects totally.
This would make things reasonably friendly to rsync. The kernel
pack has around 60M pack with 1.1M index, so everyday use would
involve incremental updates to the pack [*1*] and full download
of the index file.
[Footnote]
*1* Presumably many objects are deltified against older objects
which is suboptimal. Most likely the newer objects are accessed
far more often and they are what we would want to keep in full
not as delta. So even with this scheme we would want to have
weekly repacking. Interestingly enough, pack-objects gets the
objects via usual read_sha1_file() interface so it can produce a
new pack from an existing pack.
^ permalink raw reply
* [PATCH] Adjust fsck-cache to packed GIT and alternate object pool.
From: Junio C Hamano @ 2005-06-28 8:49 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <Pine.LNX.4.58.0506271755140.19755@ppc970.osdl.org>
>>>>> "LT" == Linus Torvalds <torvalds@osdl.org> writes:
LT> There are some other issues too, like the fact that "git-fsck-cache"
LT> doesn't know about the pack-files yet, so it will complain about missing
LT> objects etc.
And here is a patch to fix it. It is interesting to know that
the same problem existed for a long time in a different form and
nobody has complained: GIT_ALTERNATE_OBJECT_DIRECTORIES.
Maybe the alternate object pool mechanism is not so widely used
and probably not very useful for everyday use. I donno.
------------
The fsck-cache complains if objects referred to by files in
.git/refs/ or objects stored in files under .git/objects/??/ are
not found as stand-alone SHA1 files (i.e. found in alternate
object pools GIT_ALTERNATE_OBJECT_DIRECTORIES or packed archives
stored under .git/objects/pack).
Although this is a good semantics to maintain consistency of a
single .git/objects directory as a self contained set of
objects, it sometimes is useful to consider it is OK as long as
these "outside" objects are available.
This commit introduces a new flag, --standalone, to
git-fsck-cache. When it is not specified, connectivity checks
and .git/refs pointer checks are taught that it is OK when
expected objects do not exist under .git/objects/?? hierarchy
but are available from an packed archive or in an alternate
object pool.
Signed-off-by: Junio C Hamano <junkio@cox.net>
---
fsck-cache.c | 20 ++++++++++++++++----
1 files changed, 16 insertions(+), 4 deletions(-)
ea4255429bb0b4b760ba2fe327f5806d8d24d8a6
diff --git a/fsck-cache.c b/fsck-cache.c
--- a/fsck-cache.c
+++ b/fsck-cache.c
@@ -12,6 +12,7 @@
static int show_root = 0;
static int show_tags = 0;
static int show_unreachable = 0;
+static int standalone = 0;
static int keep_cache_objects = 0;
static unsigned char head_sha1[20];
@@ -25,13 +26,17 @@ static void check_connectivity(void)
struct object_list *refs;
if (!obj->parsed) {
- printf("missing %s %s\n",
- obj->type, sha1_to_hex(obj->sha1));
+ if (!standalone && has_sha1_file(obj->sha1))
+ ; /* it is in pack */
+ else
+ printf("missing %s %s\n",
+ obj->type, sha1_to_hex(obj->sha1));
continue;
}
for (refs = obj->refs; refs; refs = refs->next) {
- if (refs->item->parsed)
+ if (refs->item->parsed ||
+ (!standalone && has_sha1_file(refs->item->sha1)))
continue;
printf("broken link from %7s %s\n",
obj->type, sha1_to_hex(obj->sha1));
@@ -315,8 +320,11 @@ static int read_sha1_reference(const cha
return -1;
obj = lookup_object(sha1);
- if (!obj)
+ if (!obj) {
+ if (!standalone && has_sha1_file(sha1))
+ return 0; /* it is in pack */
return error("%s: invalid sha1 pointer %.40s", path, hexname);
+ }
obj->used = 1;
mark_reachable(obj, REACHABLE);
@@ -390,6 +398,10 @@ int main(int argc, char **argv)
keep_cache_objects = 1;
continue;
}
+ if (!strcmp(arg, "--standalone")) {
+ standalone = 1;
+ continue;
+ }
if (*arg == '-')
usage("git-fsck-cache [--tags] [[--unreachable] [--cache] <head-sha1>*]");
}
------------
^ permalink raw reply
* Re: new features in gitk
From: Greg KH @ 2005-06-28 6:22 UTC (permalink / raw)
To: Paul Mackerras; +Cc: git
In-Reply-To: <17088.31798.17291.605567@cargo.ozlabs.ibm.com>
On Tue, Jun 28, 2005 at 08:22:46AM +1000, Paul Mackerras wrote:
> I'm interested to know if people find the diff/patch generation
> options useful.
Well, I find them nice to have, in case I need to generate a real patch.
Nice job.
thanks,
greg k-h
^ permalink raw reply
* [PATCH 3/3] git-cat-file: '-s' to find out object size.
From: Junio C Hamano @ 2005-06-28 6:59 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <7vpsu7x94t.fsf@assigned-by-dhcp.cox.net>
We use sha1_object_info() now, and getting size is also trivial.
I admit that this is more of "because we can" not "because I see
immediate need for it", though.
Signed-off-by: Junio C Hamano <junkio@cox.net>
---
Documentation/git-cat-file.txt | 12 +++++++++---
cat-file.c | 13 ++++++++++---
2 files changed, 19 insertions(+), 6 deletions(-)
dcf2fc7609509985d4411fb13c6956bcf3c8560f
diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt
--- a/Documentation/git-cat-file.txt
+++ b/Documentation/git-cat-file.txt
@@ -9,12 +9,13 @@ git-cat-file - Provide content or type i
SYNOPSIS
--------
-'git-cat-file' (-t | <type>) <object>
+'git-cat-file' (-t | -s | <type>) <object>
DESCRIPTION
-----------
Provides content or type of objects in the repository. The type
-is required if '-t' is not being used to find the object type.
+is required unless '-t' is used to find the object type,
+or '-s' is used to find the object size.
OPTIONS
-------
@@ -25,6 +26,10 @@ OPTIONS
Instead of the content, show the object type identified by
<object>.
+-s::
+ Instead of the content, show the object size identified by
+ <object>.
+
<type>::
Typically this matches the real type of <object> but asking
for a type that can trivially dereferenced from the given
@@ -35,7 +40,8 @@ OPTIONS
OUTPUT
------
-If '-t' is specified, one of the <type>.
+If '-t' is specified, one of the <type>. If '-s' is specified,
+the size of the <object> in bytes.
Otherwise the raw (though uncompressed) contents of the <object> will
be returned.
diff --git a/cat-file.c b/cat-file.c
--- a/cat-file.c
+++ b/cat-file.c
@@ -13,11 +13,18 @@ int main(int argc, char **argv)
unsigned long size;
if (argc != 3 || get_sha1(argv[2], sha1))
- usage("git-cat-file [-t | tagname] <sha1>");
+ usage("git-cat-file [-t | -s | tagname] <sha1>");
- if (!strcmp("-t", argv[1])) {
+ if (!strcmp("-t", argv[1]) || !strcmp("-s", argv[1])) {
if (!sha1_object_info(sha1, type, &size)) {
- printf("%s\n", type);
+ switch (argv[1][1]) {
+ case 't':
+ printf("%s\n", type);
+ break;
+ case 's':
+ printf("%lu\n", size);
+ break;
+ }
return 0;
}
buf = NULL;
------------
^ permalink raw reply
* Re: [PATCH] Obtain sha1_file_info() for deltified pack entry properly.
From: Junio C Hamano @ 2005-06-28 6:58 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <7vpsu7x94t.fsf@assigned-by-dhcp.cox.net>
[PATCH 1/3] Obtain sha1_file_info() for deltified pack entry properly.
The initial one was not doing enough to figure things out
without uncompressing too much. It also fixes a potential
segfault resulting from missing use_packed_git() call.
We would need to introduce unuse_packed_git() call and do proper
use counting to figure out when it is safe to unmap, but
currently we do not unmap packed file yet.
Signed-off-by: Junio C Hamano <junkio@cox.net>
---
sha1_file.c | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
1 files changed, 69 insertions(+), 4 deletions(-)
d2f58b4aef500835489f30ac5df7985bc21e3c24
diff --git a/sha1_file.c b/sha1_file.c
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -601,9 +601,70 @@ void * unpack_sha1_file(void *map, unsig
return unpack_sha1_rest(&stream, hdr, *size);
}
-/* Returns 0 on fast-path success, returns 1 on deltified
- * and need to unpack to see info.
- */
+static int packed_delta_info(unsigned char *base_sha1,
+ unsigned long delta_size,
+ unsigned long left,
+ char *type,
+ unsigned long *sizep)
+{
+ unsigned char *data;
+ unsigned char delta_head[64];
+ int i;
+ unsigned char cmd;
+ unsigned long data_size, result_size, base_size, verify_base_size;
+ z_stream stream;
+ int st;
+
+ if (left < 20)
+ die("truncated pack file");
+ if (sha1_object_info(base_sha1, type, &base_size))
+ die("cannot get info for delta-pack base");
+
+ data = base_sha1 + 20;
+ data_size = left - 20;
+
+ memset(&stream, 0, sizeof(stream));
+
+ stream.next_in = data;
+ stream.avail_in = data_size;
+ stream.next_out = delta_head;
+ stream.avail_out = sizeof(delta_head);
+
+ inflateInit(&stream);
+ st = inflate(&stream, Z_FINISH);
+ inflateEnd(&stream);
+ if ((st != Z_STREAM_END) && stream.total_out != sizeof(delta_head))
+ die("delta data unpack-initial failed");
+
+ /* Examine the initial part of the delta to figure out
+ * the result size. Verify the base size while we are at it.
+ */
+ data = delta_head;
+ verify_base_size = i = 0;
+ cmd = *data++;
+ while (cmd) {
+ if (cmd & 1)
+ verify_base_size |= *data++ << i;
+ i += 8;
+ cmd >>= 1;
+ }
+
+ /* Read the result size */
+ result_size = i = 0;
+ cmd = *data++;
+ while (cmd) {
+ if (cmd & 1)
+ result_size |= *data++ << i;
+ i += 8;
+ cmd >>= 1;
+ }
+ if (verify_base_size != base_size)
+ die("delta base size mismatch");
+
+ *sizep = result_size;
+ return 0;
+}
+
static int packed_object_info(struct pack_entry *entry,
char *type, unsigned long *sizep)
{
@@ -614,12 +675,16 @@ static int packed_object_info(struct pac
offset = entry->offset;
if (p->pack_size - 5 < offset)
die("object offset outside of pack file");
+
+ if (use_packed_git(p))
+ die("cannot map packed file");
+
pack = p->pack_base + offset;
size = (pack[1] << 24) + (pack[2] << 16) + (pack[3] << 8) + pack[4];
left = p->pack_size - offset - 5;
switch (*pack) {
case 'D':
- return 1;
+ return packed_delta_info(pack+5, size, left, type, sizep);
break;
case 'C':
strcpy(type, "commit");
------------
^ permalink raw reply
* [PATCH 2/3] git-cat-file: use sha1_object_info() on '-t'.
From: Junio C Hamano @ 2005-06-28 6:58 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <7vpsu7x94t.fsf@assigned-by-dhcp.cox.net>
When trying to find out the type of the object, there is no need
to uncompress the whole object. Just use sha1_object_info().
Signed-off-by: Junio C Hamano <junkio@cox.net>
---
cat-file.c | 10 ++++------
1 files changed, 4 insertions(+), 6 deletions(-)
fcfdd3d74d359af32dd30aa9f4fe373cebb3e232
diff --git a/cat-file.c b/cat-file.c
--- a/cat-file.c
+++ b/cat-file.c
@@ -16,13 +16,11 @@ int main(int argc, char **argv)
usage("git-cat-file [-t | tagname] <sha1>");
if (!strcmp("-t", argv[1])) {
- buf = read_sha1_file(sha1, type, &size);
- if (buf) {
- buf = type;
- size = strlen(type);
- type[size] = '\n';
- size++;
+ if (!sha1_object_info(sha1, type, &size)) {
+ printf("%s\n", type);
+ return 0;
}
+ buf = NULL;
} else {
buf = read_object_with_reference(sha1, argv[1], &size, NULL);
}
------------
^ permalink raw reply
* [PATCH] Obtain sha1_file_info() for deltified pack entry properly.
From: Junio C Hamano @ 2005-06-28 6:56 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <7v64vzyqyw.fsf_-_@assigned-by-dhcp.cox.net>
I will be sending these three patches:
[PATCH 1/3] Obtain sha1_file_info() for deltified pack entry properly.
[PATCH 2/3] git-cat-file: use sha1_object_info() on '-t'.
[PATCH 3/3] git-cat-file: '-s' to find out object size.
The first one is slightly different from what I sent earlier to
you privately. If you have already applied it, please apply the
4-liner alternate patch attached to this message on top of it
for the fix included in the one in this series (and drop the
first one, obviously).
The second and third patches fell out as a bonus while I was
debugging the sha1_file_info(). Especially the third one is in
"because we can do it so cheaply now", not "because I need to
have that feature" category, and I do not mind too much if you
drop it, but I suspect somebody may find it useful.
The "4-liner alternate patch" follows.
------------
Add missing use_packed_git() call.
The function sha1_object_info() was using packed GIT file
without making sure it is mapped, which resulted in
segfaulting.
We would need to introduce unuse_packed_git() call and do proper
use counting to figure out when it is safe to unmap, but
currently we do not unmap packed file yet.
Signed-off-by: Junio C Hamano <junkio@cox.net>
---
sha1_file.c | 4 ++++
1 files changed, 4 insertions(+), 0 deletions(-)
diff --git a/sha1_file.c b/sha1_file.c
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -675,6 +675,10 @@ static int packed_object_info(struct pac
offset = entry->offset;
if (p->pack_size - 5 < offset)
die("object offset outside of pack file");
+
+ if (use_packed_git(p))
+ die("cannot map packed file");
+
pack = p->pack_base + offset;
size = (pack[1] << 24) + (pack[2] << 16) + (pack[3] << 8) + pack[4];
left = p->pack_size - offset - 5;
------------------------------------------------
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Daniel Barkalow @ 2005-06-28 5:09 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.58.0506271910390.19755@ppc970.osdl.org>
On Mon, 27 Jun 2005, Linus Torvalds wrote:
> > > [.. git-ssh-pull hopefully working ..]
> >
> > No. The pull protocol Dan did expects to throw compressed
> > representation around on the wire (which is valid if you assume
> > uncompressed transfer) and does not use read-sha1-file --
> > write-sha1-file pair, so all three do not work.
>
> Fair enough. I'd prefer for the pull/push to push object packs around
> anyway, so there's some more work there..
It shouldn't be hard to add; the main issue is determining when
transfering a pack file is a good idea, because it probably doesn't make
sense to transfer a pack file just because the source side has an object
that the target side wants in that pack. (If you pull from someone who
packed up the whole history of everything, which you already have, into a
file with one new commit, you'd be sad to get the huge thing; you really
want a little custom (or just limited) pack file.)
The ideal thing is probably to pick up some tricks from Mercurial in
figuring out what needs to be transferred, and have the source side write
a pack file directly to the connection, which the target side would then
save directly. I never worked out exactly what those tricks were, though.
The next trick would be to put something in place of cleverly-chosen
objects to specify what pack file they're in, so that the HTTP client
could find things from a packed repository. (Or we could just have an
option to unpack post-transfer.)
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: [PATCH] Skip writing out sha1 files for objects in packed git.
From: Junio C Hamano @ 2005-06-28 3:33 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <Pine.LNX.4.58.0506271935260.19755@ppc970.osdl.org>
>>>>> "LT" == Linus Torvalds <torvalds@osdl.org> writes:
LT> If we want to expand a packed file and really write the objects to the
LT> .git/objects directories, we should just not have that packed file in the
LT> .git/objects/pack directory.
What I was aiming for was this:
(1) Introduce an interface to sha1_file.c that lets you say
"use this file as one of the packs, although it is not
under .git/objects/pack";
(2) Introduce another interface to sha1_file.c that lets you
enumerate the index entries for a given pack file.
(3) Remove the unpacking logic from unpack-object.c; instead
call the above interfaces to register the pack and
enumerate entries, and call read_sha1_file() followed by
write_sha1_file() with do_expand repeatedly.
However, the infrastructure (1) and (2) may end up being a
special case only to support unpack-object (and removing the
code duplication for unpacking), in which case what you suggest
would make more sense.
LT> And if we have a pack-file in .git/objects/ that already has
LT> the object, that may not be the _same_ pack-file that we're
LT> expanding at all, so if that pack file already has the
LT> object, then not writing it out is actually the right thing
LT> to do.
This I have to think about a bit.
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Linus Torvalds @ 2005-06-28 3:30 UTC (permalink / raw)
To: Christopher Li; +Cc: Git Mailing List
In-Reply-To: <20050627235857.GA21533@64m.dyndns.org>
On Mon, 27 Jun 2005, Christopher Li wrote:
> On Mon, Jun 27, 2005 at 06:14:40PM -0700, Linus Torvalds wrote:
> >
> > The reason? The new git understands packed files natively, which ends up
> > being a much bigger win in many many ways.
>
> Interesting. I take a look at your change, it still support delta object
> inside the pack file right? For a second I am wondering you drop the delta
> feature completely.
Deltas do exist inside pack-files, yes. They just don't exist as
independent objects any more, so you can never get into the situation that
you find a delta but you don't find the delta it points to.
Because in the pack-files, there are only deltas _within_ a pack-file. You
can't have a delta that points to outside the pack.
This means that pack-files with few objects will inevitably be larger than
they could otherwise be (ie you can never have a pack file that _only_
contains deltas to the outside world), but it's just incredibly reassuring
to me that a pack-file is always self-sufficient.
So when/if we start using pack-files for doing "git pull" etc, the
pack-file won't actually help pack things for small updates: small updates
will probably contain the whole changed file, unless the update has
several changes to the same file (which is not unusual, of course), in
which case it will only contain one version and then deltas from that.
But the savings get increasingly bigger the more history we have. That's
also why the packed git archive is about 1/14th of the size of the fully
unpacked disk usage of the git project, but a packed kernel archive "only"
achieves a packing rate of 1/5th of the fully unpacked kernel archive. The
git archive is all history, while the kernel archive just "appears", and
2/3 of the files have only one single version and thus don't delta-
compress at all.
(Another reason is probably that the kernel has bigger files, which means
that it thus has relatively less loss in filesystem block padding).
But not having any outside deltas not only makes me feel safer, it also
means that you can fully validate a pack archive consistency without even
knowing what project it is from - you can check the SHA1 results of every
file in the pack against the index of the pack, and check that the SHA1's
of the pack files themselves are valid. Again, this is just a data
_consistency_ check, of course - it means that you can validate that it
downloaded fine, and that you don't have disk corruption, but it doesn't
mean that the data isn't evil and nasty and buggy ;)
Linus
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Christopher Li @ 2005-06-27 23:58 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Git Mailing List
In-Reply-To: <Pine.LNX.4.58.0506271755140.19755@ppc970.osdl.org>
On Mon, Jun 27, 2005 at 06:14:40PM -0700, Linus Torvalds wrote:
>
> The reason? The new git understands packed files natively, which ends up
> being a much bigger win in many many ways.
Interesting. I take a look at your change, it still support delta object
inside the pack file right? For a second I am wondering you drop the delta
feature completely.
Chris
^ permalink raw reply
* Re: CAREFUL! No more delta object support!
From: Linus Torvalds @ 2005-06-28 2:48 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vslz3yzwf.fsf@assigned-by-dhcp.cox.net>
On Mon, 27 Jun 2005, Junio C Hamano wrote:
>
> By the way, you broke t/t0000 with the last commit. Now an
> empty GIT_OBJECT_DIRECTORY has 257 subdirectories.
Yup, I noticed that. Fix pushed out (along with another one that was
failing because it wanted to create the "pack" directory itself, and was
unhappy when it already existed).
Linus
^ permalink raw reply
* Re: [PATCH] Skip writing out sha1 files for objects in packed git.
From: Linus Torvalds @ 2005-06-28 2:43 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vr7eni6fy.fsf_-_@assigned-by-dhcp.cox.net>
On Mon, 27 Jun 2005, Junio C Hamano wrote:
>
> Now, there's still a misfeature there, which is that when you
> create a new object, it doesn't check whether that object
> already exists in the pack-file, so you'll end up with a few
> recent objects that you really don't need (notably tree
> objects), and this patch fixes it.
>
> Signed-off-by: Junio C Hamano <junkio@cox.net>
Actually, I don't think that "do_expand" flag should exist.
If we want to expand a packed file and really write the objects to the
.git/objects directories, we should just not have that packed file in the
.git/objects/pack directory.
And if we have a pack-file in .git/objects/ that already has the object,
that may not be the _same_ pack-file that we're expanding at all, so if
that pack file already has the object, then not writing it out is actually
the right thing to do.
That will also simplify your patch a bit. I'll fix it up.
Linus
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox