* Re: git-svn failure when symlink added in svn
From: Alexander Klink @ 2007-05-01 20:53 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Eric Wong, Seth Falcon, Alexander Klink, git
In-Reply-To: <7vmz0qcuut.fsf@assigned-by-dhcp.cox.net>
On Sun, Apr 29, 2007 at 10:08:26PM -0700, Junio C Hamano wrote:
> -- >8 --
> [PATCH] Fix symlink handling in git-svn, related to PerlIO
>
[...]
> This attempts to fix the issue by using lower level sysseek and
> sysread instead of seek and read to bypass the stdio buffer.
Works fine here, too. Thanks again for the quick response ...
Regards,
Alex
^ permalink raw reply
* Re: [PATCH] gitweb: use decode_utf8 directly
From: Ismail Dönmez @ 2007-05-01 21:12 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vhcr1obuo.fsf@assigned-by-dhcp.cox.net>
[-- Attachment #1: Type: text/plain, Size: 1100 bytes --]
On Friday 27 April 2007 22:29:03 you wrote:
> Ismail Dönmez <ismail@pardus.org.tr> writes:
> >> Which means that there is something else going on. Your change
> >> may fix what you observed (I do not doubt that it fixed what you
> >> observed for you), but without understanding what really is
> >> going on (iow, why it is a fix, when the documentation clearly
> >> indicates they should be equivalent and it should not fix
> >> anything), we cannot tell what *ELSE* we are breaking with this
> >> change.
> >
> > That might be a bug in Encode itself indeed, I will dig a bit more.
> > Thanks.
>
> Thanks.
Ok found out the reason. decode() tries to decode data that is already UTF-8
and borks.
This is from Encode.pm :
sub decode_utf8($;$) {
my ( $str, $check ) = @_;
return $str if is_utf8($str); <--- Checks if the $str is already UTF-8
if ($check) {
return decode( "utf8", $str, $check ); <--- Else do what gitweb does
[...]
So my patch is indeed correct. I attach it again for reference. Can it be
please applied?
Regards,
ismail
[-- Attachment #2: decode-utf8.patch --]
[-- Type: text/x-diff, Size: 3063 bytes --]
--- gitweb/gitweb.perl 2007-04-24 16:53:00.000000000 +0300
+++ gitweb/gitweb.perl 2007-04-24 16:54:22.000000000 +0300
@@ -566,12 +566,6 @@
return $input;
}
-# very thin wrapper for decode("utf8", $str, Encode::FB_DEFAULT);
-sub to_utf8 {
- my $str = shift;
- return decode("utf8", $str, Encode::FB_DEFAULT);
-}
-
# quote unsafe chars, but keep the slash, even when it's not
# correct, but quoted slashes look too horrible in bookmarks
sub esc_param {
@@ -596,7 +590,7 @@
my $str = shift;
my %opts = @_;
- $str = to_utf8($str);
+ $str = decode_utf8($str);
$str = $cgi->escapeHTML($str);
if ($opts{'-nbsp'}) {
$str =~ s/ / /g;
@@ -610,7 +604,7 @@
my $str = shift;
my %opts = @_;
- $str = to_utf8($str);
+ $str = decode_utf8($str);
$str = $cgi->escapeHTML($str);
if ($opts{'-nbsp'}) {
$str =~ s/ / /g;
@@ -893,7 +887,7 @@
if (length($short) < length($long)) {
return $cgi->a({-href => $href, -class => "list subject",
- -title => to_utf8($long)},
+ -title => decode_utf8($long)},
esc_html($short) . $extra);
} else {
return $cgi->a({-href => $href, -class => "list subject"},
@@ -1110,7 +1104,7 @@
if (check_export_ok("$projectroot/$path")) {
my $pr = {
path => $path,
- owner => to_utf8($owner),
+ owner => decode_utf8($owner),
};
push @list, $pr
}
@@ -1139,7 +1133,7 @@
$pr = unescape($pr);
$ow = unescape($ow);
if ($pr eq $project) {
- $owner = to_utf8($ow);
+ $owner = decode_utf8($ow);
last;
}
}
@@ -1613,7 +1607,7 @@
}
my $owner = $gcos;
$owner =~ s/[,;].*$//;
- return to_utf8($owner);
+ return decode_utf8($owner);
}
## ......................................................................
@@ -1696,7 +1690,7 @@
my $title = "$site_name";
if (defined $project) {
- $title .= " - " . to_utf8($project);
+ $title .= " - " . decode_utf8($project);
if (defined $action) {
$title .= "/$action";
if (defined $file_name) {
@@ -1969,7 +1963,7 @@
print "<div class=\"page_path\">";
print $cgi->a({-href => href(action=>"tree", hash_base=>$hb),
- -title => 'tree root'}, to_utf8("[$project]"));
+ -title => 'tree root'}, decode_utf8("[$project]"));
print " / ";
if (defined $name) {
my @dirname = split '/', $name;
@@ -2584,7 +2578,7 @@
($pr->{'age'}, $pr->{'age_string'}) = @aa;
if (!defined $pr->{'descr'}) {
my $descr = git_get_project_description($pr->{'path'}) || "";
- $pr->{'descr_long'} = to_utf8($descr);
+ $pr->{'descr_long'} = decode_utf8($descr);
$pr->{'descr'} = chop_str($descr, 25, 5);
}
if (!defined $pr->{'owner'}) {
@@ -3616,7 +3610,7 @@
$hash = git_get_head_hash($project);
}
- my $filename = to_utf8(basename($project)) . "-$hash.tar.$suffix";
+ my $filename = decode_utf8(basename($project)) . "-$hash.tar.$suffix";
print $cgi->header(
-type => "application/$ctype",
^ permalink raw reply
* Re: [PATCH] gitweb: use decode_utf8 directly
From: Junio C Hamano @ 2007-05-01 21:39 UTC (permalink / raw)
To: Ismail Dönmez; +Cc: git, Jakub Narebski
In-Reply-To: <200705020012.13302.ismail@pardus.org.tr>
Ismail Dönmez <ismail@pardus.org.tr> writes:
> Ok found out the reason. decode() tries to decode data that is already UTF-8
> and borks.
>
> This is from Encode.pm :
>
> sub decode_utf8($;$) {
> my ( $str, $check ) = @_;
> return $str if is_utf8($str); <--- Checks if the $str is already UTF-8
> if ($check) {
> return decode( "utf8", $str, $check ); <--- Else do what gitweb does
> [...]
>
> So my patch is indeed correct.
Ok, I think that makes it an improvement from the current code,
so I'd apply.
But at the same time I wonder why should the callers be feeding
an already decoded string to to_utf8(). It might be that some
callers needs fixing.
^ permalink raw reply
* [PATCH] Teach import-tars about GNU tar's @LongLink extension.
From: Johannes Schindelin @ 2007-05-01 21:42 UTC (permalink / raw)
To: git, spearce
This extension allows GNU tar to process file names in excess of the 100
characters defined by the original tar standard. It does this by faking a
file, named '././@LongLink' containing the true file name, and then adding
the file with a truncated name. The idea is that tar without this
extension will write out a file with the long file name, and write the
contents into a file with truncated name.
Unfortunately, GNU tar does a lousy job at times. When truncating results
in a _directory_ name, it will happily use _that_ as a truncated name for
the file.
An example where this actually happens is gcc-4.1.2, where the full path
of the file WeThrowThisExceptionHelper.java truncates _exactly_ before the
basename. So, we have to support that ad-hoc extension.
This bug was noticed by Chris Riddoch on IRC.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
Does anybody know why they decided to limit the LongLink to 256
characters, when 512 bytes are wasted _anyway_?
BTW I am a little too tired to find out why import-tars failed so
badly without this patch. It _should_ have behaved like tar
without that extension. Instead, it tripped completely, losing the
correct block boundary.
contrib/fast-import/import-tars.perl | 19 +++++++++++++++++++
1 files changed, 19 insertions(+), 0 deletions(-)
diff --git a/contrib/fast-import/import-tars.perl b/contrib/fast-import/import-tars.perl
index d2363a4..a43b2c5 100755
--- a/contrib/fast-import/import-tars.perl
+++ b/contrib/fast-import/import-tars.perl
@@ -52,6 +52,25 @@ foreach my $tar_file (@ARGV)
Z8 Z1 Z100 Z6
Z2 Z32 Z32 Z8 Z8 Z*', $_;
last unless $name;
+ if ($name eq '././@LongLink') {
+ # GNU tar extension
+ if (read(I, $_, 512) != 512) {
+ die ('Short archive');
+ }
+ $name = unpack 'Z257', $_;
+ next unless $name;
+
+ my $dummy;
+ if (read(I, $_, 512) != 512) {
+ die ('Short archive');
+ }
+ ($dummy, $mode, $uid, $gid, $size, $mtime,
+ $chksum, $typeflag, $linkname, $magic,
+ $version, $uname, $gname, $devmajor, $devminor,
+ $prefix) = unpack 'Z100 Z8 Z8 Z8 Z12 Z12
+ Z8 Z1 Z100 Z6
+ Z2 Z32 Z32 Z8 Z8 Z*', $_;
+ }
next if $name =~ m{/\z};
$mode = oct $mode;
$size = oct $size;
--
1.5.2.rc1.2428.g1b62-dirty
^ permalink raw reply related
* Re: [PATCH] gitweb: use decode_utf8 directly
From: Ismail Dönmez @ 2007-05-01 21:44 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, Jakub Narebski
In-Reply-To: <7v8xc85ill.fsf@assigned-by-dhcp.cox.net>
On Wednesday 02 May 2007 00:39:34 you wrote:
> Ismail Dönmez <ismail@pardus.org.tr> writes:
> > Ok found out the reason. decode() tries to decode data that is already
> > UTF-8 and borks.
> >
> > This is from Encode.pm :
> >
> > sub decode_utf8($;$) {
> > my ( $str, $check ) = @_;
> > return $str if is_utf8($str); <--- Checks if the $str is already
> > UTF-8 if ($check) {
> > return decode( "utf8", $str, $check ); <--- Else do what gitweb
> > does [...]
> >
> > So my patch is indeed correct.
>
> Ok, I think that makes it an improvement from the current code,
> so I'd apply.
>
> But at the same time I wonder why should the callers be feeding
> an already decoded string to to_utf8(). It might be that some
> callers needs fixing.
FWIW it was passing my name "İsmail Dönmez" based on user info I guess.
Regards,
ismail
^ permalink raw reply
* Re: [PATCH] gitweb: use decode_utf8 directly
From: Ismail Dönmez @ 2007-05-01 21:48 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, Jakub Narebski
In-Reply-To: <200705020044.47171.ismail@pardus.org.tr>
On Wednesday 02 May 2007 00:44:46 you wrote:
> On Wednesday 02 May 2007 00:39:34 you wrote:
> > Ismail Dönmez <ismail@pardus.org.tr> writes:
> > > Ok found out the reason. decode() tries to decode data that is already
> > > UTF-8 and borks.
> > >
> > > This is from Encode.pm :
> > >
> > > sub decode_utf8($;$) {
> > > my ( $str, $check ) = @_;
> > > return $str if is_utf8($str); <--- Checks if the $str is already
> > > UTF-8 if ($check) {
> > > return decode( "utf8", $str, $check ); <--- Else do what gitweb
> > > does [...]
> > >
> > > So my patch is indeed correct.
> >
> > Ok, I think that makes it an improvement from the current code,
> > so I'd apply.
> >
> > But at the same time I wonder why should the callers be feeding
> > an already decoded string to to_utf8(). It might be that some
> > callers needs fixing.
>
> FWIW it was passing my name "İsmail Dönmez" based on user info I guess.
I guess its line 1116:
if (check_export_ok("$projectroot/$path")) {
my $pr = {
path => $path,
owner => to_utf8($owner), <---- Here
};
My system is configured for UTF-8 so $owner will be UTF-8 but in some systems
it might not be so I don't think there is anything to fix here.
Regards,
ismail
^ permalink raw reply
* Re: "Producting Open Source Software" book and distributed SCMs
From: Jakub Narebski @ 2007-05-01 18:30 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: git
In-Reply-To: <Pine.LNX.4.64.0705011057130.29859@racer.site>
Hi
On Thursday, 1 May 2007, Johannes Schindelin wrote:
> On Mon, 30 Apr 2007, Jakub Narebski wrote:
>
>> Linus has said that fully distributed SCM improves forkability:
>>
>> [...]
>>
>> I think that "forking" is what keeps people honest. The _biggest_
>> downside with CVS is actually that a central repository gets so much
>> _political_ clout, that it's effectively impossible to fork the
>> project: [...]
>>
>> According to "Producting Open Source Software" it is very important
>> feature for an OSS project.
>>
>> [...]
>>
>> Because a fork is bad for everyone (for reasons examined in detail in
>> the section called "Forks" in Chapter 8, Managing Volunteers,
>> http://producingoss.com/producingoss.html#forks), the more serious the
>> threat of a fork becomes, the more willing people are to compromise to
>> avoid it.
>
> This is a lousy argument, IMHO.
>
> Why are forks bad? They are not. But if you "learnt" that merges are hard,
> they are.
>
> It is a pity that so many people were trained in CVS, and keep thinking
> some of the lectures were true, when they are no longer.
>
> Forks are good. In fact, we all "forked" with CVS as soon as we began
> hacking. Everybody who claims to never have started over from a fresh
> checkout, or from an "update -C"ed state, is probably lying, or a bad
> developer. Thinking about it, I believe that the difference between
> forking and branching is philosophical, not technical. You can always
> merge a fork.
IIRC Compiz and Beryl (fork of Compiz) plan to be merged. Both projects
use git as SCM. We will see how this "merge a fork" will work.
In "Producting Open Source Software" Karl Fogel gives an example of
GCC/EGCS fork, which resulted in "fast forward" merge (EGCS which was
fork of GCC, became next version of GCC). Similar example is XFree86/X.Org
fork; Linux distributions went from packaging XFree86 to packaging X.Org.
But for example GNU Emacs / XEmacs fork will never be merged, I think.
So not always you can merge a fork - you can try, unless codebase diverged
too much.
>> Is distributed SCM better geared towards "benovolent dictator"
>> model than "consensus-based democracy" model, as described in
>> OSSbook?
>
> Not at all. I think the best example is kernel.org, where you find
> tons of forks. IMHO it is really helping the benevolent dictator cave
> into the consensus-based model, since forks can be preferred at any
> time. Hey, even switching from one to another upstream is just a
> git-pull away!
What is or is not a fork is a bit blurry in the world of distributed
version control systems. Is a clone of repository a fork? I think that
everybody would agree that it is not. Is for example *-mm tree a fork?
I'd say not. But I'd say that Beryl is a fork of Compiz...
--
Jakub Narebski
ShadeHawk on #git
Poland
^ permalink raw reply
* Git benchmarks at OpenOffice.org wiki
From: Jakub Narebski @ 2007-05-01 21:46 UTC (permalink / raw)
To: git; +Cc: releases, Jan Holesovsky
OpenOffice.org is looking for a new SCM (Software Configuration
Management) tool, or at least was on Friday, 19 Jan 2007;
see: http://blogs.sun.com/GullFOSS/entry/openoffice_org_scm
One of the SCMs considered is Git. One of others is Subversion.
There is a functional git tree with the entire OOo history for testing
purposes that can be found at: http://go-oo.org/git.
What I am concerned about is some of git benchmark results at Git page
on OpenOffice.org wiki:
http://wiki.services.openoffice.org/wiki/Git#Comparison
Actually it is comparison with CVS and Subversion, although most
benchmarks are done only for git.
In 'Size of data on the server' git has CVS beat hands down: 1.3G vs
8.5G for sources, 591M vs 1.1G for third party. I think it is similar
for Subversion. I hope that repository is fully packed: IIRC the Mozilla
CVS repository import was about 0.6GB pack file, not 1.3GB.
The problem is with 'Size of checkout': to start working in repository
one needs 1.4G (sources) and 98M (third party) for CVS checkout (it is
1.5G for sources for Subversion checkout). Ordinary for distributed SCM
you would need size of repository + size of sources (working area),
which is 2.8G for sources and 688M for third party stuff files you can
hack on + the history]. This makes some prefer to go centralized SCM
route, i.e. Subversion as replacement for CVS (+ CWS, ChildWorkSpace).
What might help here is splitting repository into current (e.g. from
OOo 2.0) and historical part, and / or using shallow clone. Implementing
partial checkouts, i.e. checking out only part of working area (and
using 'theirs' strategy for merging not-checked-out part for merges)
would help. Splitting repository into submodules, and submodule
support -- it depends on organization of OOo sources, would certainly
help for third party stuff repository.
'Checkout time' (which should be renamed to 'Initial checkout time'),
in which git also loses with 130 minutes (Linux, 2MBit DSL) [from
go-oo.org], 100min (Linux, 2MBit DSL, Wireless, no proxy) [from
go-oo.org] versus 117 minutes (Linux, 2MBit DSL), 26 minutes (Linux,
2MBit DSL, with compression (-z 6)) for CVS, and 60 Minutes (Windows,
34Mbit Line) for Subversion, would also be helped by the above.
What I'm really concerned about is branch switch and merging branches,
when one of the branches is an old one (e.g. unxsplash branch), which
takes 3min (!) according to the benchmark. 13-25sec for commit is also
bit long, but BRANCH SWITCHING which takes 3 MINUTES!? There is no
comparison benchmark for CVS or Subversion, though...
Comparison / benchmark lacks some crucial info, like what computer was
used (CPU, RAM, HDD), what filesystem was used, git version etc. It
does have commands used for tests (benchmarks).
Could you confirm (or deny) those results? go-oo.org uses git 1.4.3.4;
was there some improvement or bugfix related to the speed of checkout?
--
Jakub Narebski
ShadeHawk on #git
Poland
^ permalink raw reply
* Re: svn:externals using git submodules
From: Linus Torvalds @ 2007-05-01 22:19 UTC (permalink / raw)
To: Chris Shoemaker; +Cc: Andy Parkins, git, Shawn O. Pearce
In-Reply-To: <20070501202356.GA25531@pe.Belkin>
On Tue, 1 May 2007, Chris Shoemaker wrote:
>
> Making git-svn handle svn:externals with specified revisions would be
> _quite_ useful. There's a special-case of this that I use personally:
> svn:externals that point to other paths (and other revisions) of the
> parent repo.
Side note: even _without_ a specified revision, I think it's quite sane to
have the rule that a submodule hash of all zeroes is "unversioned".
Such a submodule is still _useful_: while the tree itself contains no
information (and it SHOULD NOT do so, since the actual location of the
external module may not be globally stable or visible!), it would
basically act like subversion externals together with the ".gitmodules"
file that contains that information.
So while the git submodule thing was designed to specify specific
revisions, there's nothing that really technically _requires_ it. The
exact SHA1 details in the submodule link are going to be up to the
higher-level user anyway.
(Of course, if you actually have a "all zeroes" gitlink entry, and then
have a checked-out git tree at that entry, "git status" and "git diff"
would show it as needing update. I think that's _correct_, but if we want
to shut it up for the special case of all-zero SHA1, we trivially could).
But while I'm encouraged that the whole gitlink thing seems to be working
for Andy, and some others are playing with it too, I'm also a bit
discouraged by the fact that there hasn't been any noise or work on the
porcelain side. I was obviously optimistic and hoping we'd see support in
checkout/diff, but I haven't heard anybody talk about actually
implementing .gitmodules and the porcelain support that uses them..
Linus
^ permalink raw reply
* Re: "Producting Open Source Software" book and distributed SCMs
From: Jakub Narebski @ 2007-05-01 22:27 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <alpine.LFD.0.98.0705010829180.3808@woody.linux-foundation.org>
Linus Torvalds wrote:
> And the nice thing about this is that if some subsystem needs to make
> trivial changes to another subsystem, they don't need to ask for
> permission. They just do them, AND THEN THEY EXPLAIN THEM! And if they
> really were trivial and obvious (and that's almost always the case), they
> just get pulled normally. No special dispensation.
Actually Karl Fogel wrote in "Producting Open Source Software" that he
recommends and uses 'soft' partial commit access; it means that committing
is restricted to a part of project for some by a guideline, but is not
enforced by the tool (by SCM).
P.S. I recommend actually reading the book (at http://producingoss.com)
instead of relying on my understanding of it.
--
Jakub Narebski
ShadeHawk on #git
Poland
^ permalink raw reply
* Re: Git benchmarks at OpenOffice.org wiki
From: Junio C Hamano @ 2007-05-01 22:27 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git, releases, Jan Holesovsky
In-Reply-To: <200705012346.14997.jnareb@gmail.com>
Jakub Narebski <jnareb@gmail.com> writes:
> What might help here is splitting repository into current (e.g. from
> OOo 2.0) and historical part, and / or using shallow clone.
Yes, depending on where you cut off and how reasonable the
project history is.
> Implementing
> partial checkouts, i.e. checking out only part of working area (and
> using 'theirs' strategy for merging not-checked-out part for merges)
> would help.
Partial checkouts, perhaps, "theirs", NO.
Consider that you are working on the tip with partial checkout.
Somebody has a bugfix that is applicable to all of ancient, old,
maintenance and current codebase. Naturally you would want the
bugfix to be applied to ancient, merge it to old, and then
maintenance and then current (the last one is what you are
working on).
What happens if you actually pull ancient when you are partially
checked out and use "theirs"?
> Splitting repository into submodules, and submodule
> support -- it depends on organization of OOo sources, would certainly
> help for third party stuff repository.
This is probably the most sane way.
^ permalink raw reply
* Re: This seems somewhat less that ideal
From: A Large Angry SCM @ 2007-05-01 22:32 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vmz0p88u0.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano wrote:
> A Large Angry SCM <gitzilla@gmail.com> writes:
>
>> remote: Generating pack...
>> remote: Done counting 46473 objects.
>> remote: Deltifying 46473 objects.
>> remote: 100% (46473/46473) done
>> Indexing 46473 objects...
>> remote: Total 46473 (delta 32624), reused 46043 (delta 32299)
>> 100% (46473/46473) done
>> Resolving 32624 deltas...
>> 100% (32624/32624) done
>
> Indeed it seems somewhat less than ideal. Funny thing is that
> it does not reproduce.
Not hear either, but I'm not able to recreate the precise conditions again.
^ permalink raw reply
* Re: svn:externals using git submodules
From: Junio C Hamano @ 2007-05-01 22:37 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Chris Shoemaker, Andy Parkins, git, Shawn O. Pearce
In-Reply-To: <alpine.LFD.0.98.0705011512300.3808@woody.linux-foundation.org>
Linus Torvalds <torvalds@linux-foundation.org> writes:
> On Tue, 1 May 2007, Chris Shoemaker wrote:
>>
>> Making git-svn handle svn:externals with specified revisions would be
>> _quite_ useful. There's a special-case of this that I use personally:
>> svn:externals that point to other paths (and other revisions) of the
>> parent repo.
>
> Side note: even _without_ a specified revision, I think it's quite sane to
> have the rule that a submodule hash of all zeroes is "unversioned".
Yup.
> But while I'm encouraged that the whole gitlink thing seems to be working
> for Andy, and some others are playing with it too, I'm also a bit
> discouraged by the fact that there hasn't been any noise or work on the
> porcelain side. I was obviously optimistic and hoping we'd see support in
> checkout/diff, but I haven't heard anybody talk about actually
> implementing .gitmodules and the porcelain support that uses them..
The thing is, almost all the core git people happen to be busy
at the same time at this moment. Johannes has just moved, Shawn
and I are deep in day-jobs to the neck, ...
Don't worry, it eventually will come.
^ permalink raw reply
* Re: "Producting Open Source Software" book and distributed SCMs
From: Linus Torvalds @ 2007-05-01 22:45 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <200705020027.41316.jnareb@gmail.com>
On Wed, 2 May 2007, Jakub Narebski wrote:
>
> Actually Karl Fogel wrote in "Producting Open Source Software" that he
> recommends and uses 'soft' partial commit access; it means that committing
> is restricted to a part of project for some by a guideline, but is not
> enforced by the tool (by SCM).
Oh, absolutely. Except that really does require a lot of trust up front,
which is the problem with commit access to begin with - you automatically
have a very clear (and *big*) difference between insiders and outsiders,
and there is no "gradual" way to move from one to the other.
So yes, for practical reasons, "commit access" really is almost always an
all-or-nothing thing for most centralized setups, because nothing else
really works. And when it isn't, it's just a horrible horrible pain in the
*ss.
What people do instead of commit access is to set up triggers to notify
people about certain subsystems being modified. Which is a good idea, but
it's really a totally different thing.
> P.S. I recommend actually reading the book (at http://producingoss.com)
> instead of relying on my understanding of it.
It actually looks like a fine book, even though I think Karl is totally
off in not seeing the big difference between centralized and distributed.
I saw it at the local Borders, and considered buying it. I didn't even
realize that it apparently is downloadable too.
And it talks about a lot of other things than just SCM's.
Linus
^ permalink raw reply
* Re: "Producting Open Source Software" book and distributed SCMs
From: Linus Torvalds @ 2007-05-01 23:13 UTC (permalink / raw)
To: Jakub Narebski; +Cc: Johannes Schindelin, git
In-Reply-To: <200705012030.11747.jnareb@gmail.com>
On Tue, 1 May 2007, Jakub Narebski wrote:
>
> In "Producting Open Source Software" Karl Fogel gives an example of
> GCC/EGCS fork, which resulted in "fast forward" merge (EGCS which was
> fork of GCC, became next version of GCC).
The egcs fork was a total disaster, and a big part of that was CVS and the
tight control of the gcc tree.
It took _years_ for people to get so fed up with the gcc maintenance that
the egcs tree happened at all, and it was a prime example of how *painful*
CVS makes this, and how it allowed the gcc maintainers to do a really bad
job, and ignore a whole lot of major problems simply because the whole gcc
setup was so hard to get into.
So yes, the egcs fork is a great example. It was not only a required (and
very good) fork, but it is _also_ an example of a setup where all the
infrastructure made the fork take a lot longer to materialize and be a lot
more painful than it should have been.
> But for example GNU Emacs / XEmacs fork will never be merged, I think.
> So not always you can merge a fork - you can try, unless codebase diverged
> too much.
In all honesty, I don't think any tools would help there. Git can make
merging easier, but it cannot solve the fundamental differences in
personality and it can't help with ten years of differences. Git tries to
make merging easy by making it happen all the time, and thus the git
merge capability really depend on changing the *model*. But git cannot
really help you all that much if you have a decade of split, and the
codebases just don't look similar any more..
(Not entirely true: git obviously does make merging easier, since people
have piped up to say that they imported branches from SVN just to merge
them in git and push the result back to SVN. So git _does_ help on the
pure technical side too, but I think the even more important part is how
git tries to encourage the model to be that one or both sides just merge
often enough that the merges _stay_ easy).
> What is or is not a fork is a bit blurry in the world of distributed
> version control systems. Is a clone of repository a fork? I think that
> everybody would agree that it is not. Is for example *-mm tree a fork?
> I'd say not. But I'd say that Beryl is a fork of Compiz...
Well, the -mm tree is a fork, but perhaps the difference is that the
_intention_ is to merge back.
We've had "real forks" in the kernel community too. Vendor branches for a
while tended to be real forks - not because the vendors didn't want to
merge back, but simply because they didn't have the capability and
commitment to do so. That's changed, partly because 2.4->2.6 was so
painful for some of them.
And the VM people have had real forks. The -aa tree wasa real fork in the
2.4.x timeframe.
So I think the reason kernel people don't really think about "-mm" as a
fork is that we've tended to be pretty amicable about the forks, whatever
the intention was. I personally encourage them, for example, in ways that
most other bigger open source projects do not. That makes it easier
psychologically to fork, but more importantly, it also makes it easier to
join back again, because there was generally no hard feelings, just
differences of opinion on technical matters that didn't get to be _too_
personal.
So to _me_, the big issue is not so much forking, but joining it all back
(ie merging). Forking should be trivial, and not even worthy of any real
discussion. It should be a daily event, and sure, you'd expect the small
forks to heavily outnumber the big ones, but none of that really matters
if you just consider forking to not be a big deal - and always realize
that joining back is where the interesting stuff happens!
Linus
^ permalink raw reply
* Re: svn:externals using git submodules
From: Shawn O. Pearce @ 2007-05-01 23:16 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Linus Torvalds, Chris Shoemaker, Andy Parkins, git
In-Reply-To: <7vzm4o41bz.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano <junkio@cox.net> wrote:
> Linus Torvalds <torvalds@linux-foundation.org> writes:
> > But while I'm encouraged that the whole gitlink thing seems to be working
> > for Andy, and some others are playing with it too, I'm also a bit
> > discouraged by the fact that there hasn't been any noise or work on the
> > porcelain side. I was obviously optimistic and hoping we'd see support in
> > checkout/diff, but I haven't heard anybody talk about actually
> > implementing .gitmodules and the porcelain support that uses them..
>
> The thing is, almost all the core git people happen to be busy
> at the same time at this moment. Johannes has just moved, Shawn
> and I are deep in day-jobs to the neck, ...
Heh, very true. My short-term focus (this week) is git-gui,
then pack v4, and probably while working on pack v4 start at least
playing with Linus' gitlink thing and prototyping porcleain over it.
The gitlink work interests me, and I want to spend time on it,
but as Junio said, I'm overbooked...
--
Shawn.
^ permalink raw reply
* [PATCH 0/5] git-repack --max-pack-size
From: Dana How @ 2007-05-02 0:02 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Git Mailing List, danahow
The three most common ways of making large packfiles
are git-fast-import, the first git-repack, or git-repack -a.
The first already supports a "--max-pack-size=N" option,
which limits the resulting packfiles to N megabytes.
This patchset adds the same option, with the same
behavior, to git-repack to handle the other two cases.
This iteration fixes the most recent issues,
including operation without implying -f/--no-reuse-delta.
^ permalink raw reply
* [PATCH 1/5] git-repack --max-pack-size: alter sha1close() 3rd argument
From: Dana How @ 2007-05-02 0:03 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Git Mailing List, danahow
update=0 suppressed writing the final SHA-1 but was not used.
Now final=0 suppresses SHA-1 finalization, SHA-1 writing,
and closing -- in other words, only flush the buffer.
Signed-off-by: Dana L. How <danahow@gmail.com>
---
csum-file.c | 8 +++++---
1 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/csum-file.c b/csum-file.c
index 7c806ad..9913cb5 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -29,18 +29,20 @@ static void sha1flush(struct sha1file *f, unsigned int count)
}
}
-int sha1close(struct sha1file *f, unsigned char *result, int update)
+int sha1close(struct sha1file *f, unsigned char *result, int final)
{
unsigned offset = f->offset;
if (offset) {
SHA1_Update(&f->ctx, f->buffer, offset);
sha1flush(f, offset);
+ f->offset = 0;
}
+ if (!final)
+ return 0; /* only want to flush (no checksum write, no close) */
SHA1_Final(f->buffer, &f->ctx);
if (result)
hashcpy(result, f->buffer);
- if (update)
- sha1flush(f, 20);
+ sha1flush(f, 20);
if (close(f->fd))
die("%s: sha1 file error on close (%s)", f->name, strerror(errno));
free(f);
--
1.5.2.rc0.789.gd951
^ permalink raw reply related
* [PATCH 2/5] git-repack --max-pack-size: new file statics and code restructuring
From: Dana How @ 2007-05-02 0:05 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Git Mailing List, danahow
Add "pack_size_limit", the limit specified by --max-pack-size,
"written_list", the list of objects written to the current pack,
and "nr_written", the number of objects in written_list.
Put "base_name" at file scope again and add forward declarations.
Move write_index_file() call from cnd_pack_objects() to
write_pack_file() since only the latter will know how
many times to call write_index_file().
Signed-off-by: Dana L. How <danahow@gmail.com>
---
builtin-pack-objects.c | 68 +++++++++++++++++++++++++----------------------
1 files changed, 36 insertions(+), 32 deletions(-)
diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index b827627..bc45ca6 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -52,7 +52,8 @@ struct object_entry {
* nice "minimum seek" order.
*/
static struct object_entry *objects;
-static uint32_t nr_objects, nr_alloc, nr_result;
+static struct object_entry **written_list;
+static uint32_t nr_objects, nr_alloc, nr_result, nr_written;
static int non_empty;
static int no_reuse_delta;
@@ -61,9 +62,11 @@ static int incremental;
static int allow_ofs_delta;
static const char *pack_tmp_name, *idx_tmp_name;
static char tmpname[PATH_MAX];
+static const char *base_name;
static unsigned char pack_file_sha1[20];
static int progress = 1;
static int window = 10;
+static uint32_t pack_size_limit;
static int pack_to_stdout;
static int num_preferred_base;
static struct progress progress_state;
@@ -559,7 +562,11 @@ static off_t write_one(struct sha1file *f,
return offset + size;
}
-static off_t write_pack_file(void)
+/* forward declarations for write_pack_file */
+static void write_index_file(off_t last_obj_offset, unsigned char *sha1);
+static int adjust_perm(const char *path, mode_t mode);
+
+static void write_pack_file(void)
{
uint32_t i;
struct sha1file *f;
@@ -603,7 +610,32 @@ static off_t write_pack_file(void)
die("wrote %u objects while expecting %u", written, nr_result);
sha1close(f, pack_file_sha1, 1);
- return last_obj_offset;
+ if (!pack_to_stdout) {
+ unsigned char object_list_sha1[20];
+ mode_t mode = umask(0);
+
+ umask(mode);
+ mode = 0444 & ~mode;
+
+ write_index_file(last_obj_offset, object_list_sha1);
+ snprintf(tmpname, sizeof(tmpname), "%s-%s.pack",
+ base_name, sha1_to_hex(object_list_sha1));
+ if (adjust_perm(pack_tmp_name, mode))
+ die("unable to make temporary pack file readable: %s",
+ strerror(errno));
+ if (rename(pack_tmp_name, tmpname))
+ die("unable to rename temporary pack file: %s",
+ strerror(errno));
+ snprintf(tmpname, sizeof(tmpname), "%s-%s.idx",
+ base_name, sha1_to_hex(object_list_sha1));
+ if (adjust_perm(idx_tmp_name, mode))
+ die("unable to make temporary index file readable: %s",
+ strerror(errno));
+ if (rename(idx_tmp_name, tmpname))
+ die("unable to rename temporary index file: %s",
+ strerror(errno));
+ puts(sha1_to_hex(object_list_sha1));
+ }
}
static int sha1_sort(const void *_a, const void *_b)
@@ -1586,8 +1618,6 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
int use_internal_rev_list = 0;
int thin = 0;
uint32_t i;
- off_t last_obj_offset;
- const char *base_name = NULL;
const char **rp_av;
int rp_ac_alloc = 64;
int rp_ac;
@@ -1738,33 +1768,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
fprintf(stderr, "Result has %u objects.\n", nr_result);
if (nr_result)
prepare_pack(window, depth);
- last_obj_offset = write_pack_file();
- if (!pack_to_stdout) {
- unsigned char object_list_sha1[20];
- mode_t mode = umask(0);
-
- umask(mode);
- mode = 0444 & ~mode;
-
- write_index_file(last_obj_offset, object_list_sha1);
- snprintf(tmpname, sizeof(tmpname), "%s-%s.pack",
- base_name, sha1_to_hex(object_list_sha1));
- if (adjust_perm(pack_tmp_name, mode))
- die("unable to make temporary pack file readable: %s",
- strerror(errno));
- if (rename(pack_tmp_name, tmpname))
- die("unable to rename temporary pack file: %s",
- strerror(errno));
- snprintf(tmpname, sizeof(tmpname), "%s-%s.idx",
- base_name, sha1_to_hex(object_list_sha1));
- if (adjust_perm(idx_tmp_name, mode))
- die("unable to make temporary index file readable: %s",
- strerror(errno));
- if (rename(idx_tmp_name, tmpname))
- die("unable to rename temporary index file: %s",
- strerror(errno));
- puts(sha1_to_hex(object_list_sha1));
- }
+ write_pack_file();
if (progress)
fprintf(stderr, "Total %u (delta %u), reused %u (delta %u)\n",
written, written_delta, reused, reused_delta);
--
1.5.2.rc0.789.gd951
^ permalink raw reply related
* [PATCH 3/5] git-repack --max-pack-size: write_{object,one}() respect pack limit
From: Dana How @ 2007-05-02 0:06 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Git Mailing List, danahow
With --max-pack-size, generate the appropriate write limit
for each object and check against it before each group of writes.
Update delta usability rules to handle base being in a previously-
written pack. Inline sha1write_compress() so we know the
exact size of the written data when it needs to be compressed.
Detect and return write "failure".
Signed-off-by: Dana L. How <danahow@gmail.com>
---
builtin-pack-objects.c | 134 ++++++++++++++++++++++++++++++++++++++---------
1 files changed, 108 insertions(+), 26 deletions(-)
diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index bc45ca6..dba5681 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -399,16 +399,21 @@ static int revalidate_loose_object(struct object_entry *entry,
}
static unsigned long write_object(struct sha1file *f,
- struct object_entry *entry)
+ struct object_entry *entry,
+ off_t write_offset)
{
unsigned long size;
enum object_type type;
void *buf;
unsigned char header[10];
+ unsigned char dheader[10];
unsigned hdrlen;
off_t datalen;
enum object_type obj_type;
int to_reuse = 0;
+ /* write limit if limited packsize and not first object */
+ unsigned long limit = pack_size_limit && nr_written ?
+ pack_size_limit - write_offset : 0;
if (!pack_to_stdout)
crc32_begin(f);
@@ -417,7 +422,10 @@ static unsigned long write_object(struct sha1file *f,
if (! entry->in_pack)
to_reuse = 0; /* can't reuse what we don't have */
else if (obj_type == OBJ_REF_DELTA || obj_type == OBJ_OFS_DELTA)
- to_reuse = 1; /* check_object() decided it for us */
+ /* check_object() decided it for us ... */
+ to_reuse = !pack_size_limit ||
+ (entry->delta->offset && entry->delta->offset != (off_t)-1);
+ /* ... but pack split may override that */
else if (obj_type != entry->in_pack_type)
to_reuse = 0; /* pack has delta which is unusable */
else if (entry->delta)
@@ -436,6 +444,10 @@ static unsigned long write_object(struct sha1file *f,
if (revalidate_loose_object(entry, map, mapsize))
die("corrupt loose object %s",
sha1_to_hex(entry->sha1));
+ if (limit && mapsize + 20 >= limit) {
+ munmap(map, mapsize);
+ return 0;
+ }
sha1write(f, map, mapsize);
munmap(map, mapsize);
written++;
@@ -447,24 +459,58 @@ static unsigned long write_object(struct sha1file *f,
}
if (!to_reuse) {
+ z_stream stream;
+ unsigned long maxsize;
+ void *out;
+ /* no if no delta */
+ int usable_delta = !entry->delta ? 0 :
+ /* yes if unlimited packfile */
+ !pack_size_limit ? 1 :
+ /* no if base written to previous pack */
+ entry->delta->offset == (off_t)-1 ? 0 :
+ /* otherwise double-check written to this
+ * pack, like we do below
+ */
+ entry->delta->offset ? 1 : 0;
buf = read_sha1_file(entry->sha1, &type, &size);
if (!buf)
die("unable to read %s", sha1_to_hex(entry->sha1));
if (size != entry->size)
die("object %s size inconsistency (%lu vs %lu)",
sha1_to_hex(entry->sha1), size, entry->size);
- if (entry->delta) {
+ if (usable_delta) {
buf = delta_against(buf, size, entry);
size = entry->delta_size;
obj_type = (allow_ofs_delta && entry->delta->offset) ?
OBJ_OFS_DELTA : OBJ_REF_DELTA;
+ } else {
+ /*
+ * recover real object type in case
+ * check_object() wanted to re-use a delta,
+ * but we couldn't since base was in previous split pack
+ */
+ obj_type = type;
}
+ /* compress the data to store and put compressed length in datalen */
+ memset(&stream, 0, sizeof(stream));
+ deflateInit(&stream, zlib_compression_level);
+ maxsize = deflateBound(&stream, size);
+ out = xmalloc(maxsize);
+ /* Compress it */
+ stream.next_in = buf;
+ stream.avail_in = size;
+ stream.next_out = out;
+ stream.avail_out = maxsize;
+ while (deflate(&stream, Z_FINISH) == Z_OK)
+ /* nothing */;
+ deflateEnd(&stream);
+ datalen = stream.total_out;
+ deflateEnd(&stream);
/*
* The object header is a byte of 'type' followed by zero or
* more bytes of length.
*/
hdrlen = encode_header(obj_type, size, header);
- sha1write(f, header, hdrlen);
if (obj_type == OBJ_OFS_DELTA) {
/*
@@ -473,21 +519,41 @@ static unsigned long write_object(struct sha1file *f,
* base from this object's position in the pack.
*/
off_t ofs = entry->offset - entry->delta->offset;
- unsigned pos = sizeof(header) - 1;
- header[pos] = ofs & 127;
+ unsigned pos = sizeof(dheader) - 1;
+ dheader[pos] = ofs & 127;
while (ofs >>= 7)
- header[--pos] = 128 | (--ofs & 127);
- sha1write(f, header + pos, sizeof(header) - pos);
- hdrlen += sizeof(header) - pos;
+ dheader[--pos] = 128 | (--ofs & 127);
+ if (limit && hdrlen + sizeof(dheader) - pos + datalen + 20 >= limit) {
+ free(out);
+ free(buf);
+ return 0;
+ }
+ sha1write(f, header, hdrlen);
+ sha1write(f, dheader + pos, sizeof(dheader) - pos);
+ hdrlen += sizeof(dheader) - pos;
} else if (obj_type == OBJ_REF_DELTA) {
/*
* Deltas with a base reference contain
* an additional 20 bytes for the base sha1.
*/
+ if (limit && hdrlen + 20 + datalen + 20 >= limit) {
+ free(out);
+ free(buf);
+ return 0;
+ }
+ sha1write(f, header, hdrlen);
sha1write(f, entry->delta->sha1, 20);
hdrlen += 20;
+ } else {
+ if (limit && hdrlen + datalen + 20 >= limit) {
+ free(out);
+ free(buf);
+ return 0;
+ }
+ sha1write(f, header, hdrlen);
}
- datalen = sha1write_compressed(f, buf, size);
+ sha1write(f, out, datalen);
+ free(out);
free(buf);
}
else {
@@ -502,20 +568,6 @@ static unsigned long write_object(struct sha1file *f,
reused_delta++;
}
hdrlen = encode_header(obj_type, entry->size, header);
- sha1write(f, header, hdrlen);
- if (obj_type == OBJ_OFS_DELTA) {
- off_t ofs = entry->offset - entry->delta->offset;
- unsigned pos = sizeof(header) - 1;
- header[pos] = ofs & 127;
- while (ofs >>= 7)
- header[--pos] = 128 | (--ofs & 127);
- sha1write(f, header + pos, sizeof(header) - pos);
- hdrlen += sizeof(header) - pos;
- } else if (obj_type == OBJ_REF_DELTA) {
- sha1write(f, entry->delta->sha1, 20);
- hdrlen += 20;
- }
-
offset = entry->in_pack_offset;
revidx = find_packed_object(p, offset);
datalen = revidx[1].offset - offset;
@@ -524,6 +576,29 @@ static unsigned long write_object(struct sha1file *f,
die("bad packed object CRC for %s", sha1_to_hex(entry->sha1));
offset += entry->in_pack_header_size;
datalen -= entry->in_pack_header_size;
+ if (obj_type == OBJ_OFS_DELTA) {
+ off_t ofs = entry->offset - entry->delta->offset;
+ unsigned pos = sizeof(dheader) - 1;
+ dheader[pos] = ofs & 127;
+ while (ofs >>= 7)
+ dheader[--pos] = 128 | (--ofs & 127);
+ if (limit && hdrlen + sizeof(dheader) - pos + datalen + 20 >= limit)
+ return 0;
+ sha1write(f, header, hdrlen);
+ sha1write(f, dheader + pos, sizeof(dheader) - pos);
+ hdrlen += sizeof(dheader) - pos;
+ } else if (obj_type == OBJ_REF_DELTA) {
+ if (limit && hdrlen + 20 + datalen + 20 >= limit)
+ return 0;
+ sha1write(f, header, hdrlen);
+ sha1write(f, entry->delta->sha1, 20);
+ hdrlen += 20;
+ } else {
+ if (limit && hdrlen + datalen + 20 >= limit)
+ return 0;
+ sha1write(f, header, hdrlen);
+ }
+
if (!pack_to_stdout && p->index_version == 1 &&
check_pack_inflate(p, &w_curs, offset, datalen, entry->size))
die("corrupt packed object for %s", sha1_to_hex(entry->sha1));
@@ -550,11 +625,18 @@ static off_t write_one(struct sha1file *f,
return offset;
/* if we are deltified, write out base object first. */
- if (e->delta)
+ if (e->delta) {
offset = write_one(f, e->delta, offset);
+ if (!offset)
+ return 0;
+ }
e->offset = offset;
- size = write_object(f, e);
+ size = write_object(f, e, offset);
+ if (!size) {
+ e->offset = 0;
+ return 0;
+ }
/* make sure off_t is sufficiently large not to wrap */
if (offset > offset + size)
--
1.5.2.rc0.789.gd951
^ permalink raw reply related
* [PATCH 4/5] git-repack --max-pack-size: split packs as asked by write_{object,one}
From: Dana How @ 2007-05-02 0:07 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Git Mailing List, danahow
Rewrite write_pack_file() to break to a new packfile
whenever write_object/write_one request it, and
correct the header's object count in the previous packfile.
Change write_index_file() to write an index
for just the objects in the most recent packfile.
Signed-off-by: Dana L. How <danahow@gmail.com>
---
builtin-pack-objects.c | 165 +++++++++++++++++++++++++++---------------------
1 files changed, 93 insertions(+), 72 deletions(-)
diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index dba5681..a7b19c2 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -637,6 +637,7 @@ static off_t write_one(struct sha1file *f,
e->offset = 0;
return 0;
}
+ written_list[nr_written++] = e;
/* make sure off_t is sufficiently large not to wrap */
if (offset > offset + size)
@@ -650,74 +651,102 @@ static int adjust_perm(const char *path, mode_t mode);
static void write_pack_file(void)
{
- uint32_t i;
+ uint32_t i = 0, j;
struct sha1file *f;
- off_t offset, last_obj_offset = 0;
+ off_t offset, offset_one, last_obj_offset = 0;
struct pack_header hdr;
- int do_progress = progress;
-
- if (pack_to_stdout) {
- f = sha1fd(1, "<stdout>");
- do_progress >>= 1;
- } else {
- int fd;
- snprintf(tmpname, sizeof(tmpname), "tmp_pack_XXXXXX");
- fd = mkstemp(tmpname);
- if (fd < 0)
- die("unable to create %s: %s\n", tmpname, strerror(errno));
- pack_tmp_name = xstrdup(tmpname);
- f = sha1fd(fd, pack_tmp_name);
- }
+ int do_progress = progress >> pack_to_stdout;
+ uint32_t nr_remaining = nr_result;
if (do_progress)
start_progress(&progress_state, "Writing %u objects...", "", nr_result);
+ written_list = xmalloc(nr_objects * sizeof(struct object_entry *));
- hdr.hdr_signature = htonl(PACK_SIGNATURE);
- hdr.hdr_version = htonl(PACK_VERSION);
- hdr.hdr_entries = htonl(nr_result);
- sha1write(f, &hdr, sizeof(hdr));
- offset = sizeof(hdr);
- if (!nr_result)
- goto done;
- for (i = 0; i < nr_objects; i++) {
- last_obj_offset = offset;
- offset = write_one(f, objects + i, offset);
- if (do_progress)
- display_progress(&progress_state, written);
- }
+ do {
+ if (pack_to_stdout) {
+ f = sha1fd(1, "<stdout>");
+ } else {
+ int fd;
+ snprintf(tmpname, sizeof(tmpname), "tmp_pack_XXXXXX");
+ fd = mkstemp(tmpname);
+ if (fd < 0)
+ die("unable to create %s: %s\n", tmpname, strerror(errno));
+ pack_tmp_name = xstrdup(tmpname);
+ f = sha1fd(fd, pack_tmp_name);
+ }
+
+ hdr.hdr_signature = htonl(PACK_SIGNATURE);
+ hdr.hdr_version = htonl(PACK_VERSION);
+ hdr.hdr_entries = htonl(nr_remaining);
+ sha1write(f, &hdr, sizeof(hdr));
+ offset = sizeof(hdr);
+ nr_written = 0;
+ for (; i < nr_objects; i++) {
+ last_obj_offset = offset;
+ offset_one = write_one(f, objects + i, offset);
+ if (!offset_one)
+ break;
+ offset = offset_one;
+ if (do_progress)
+ display_progress(&progress_state, written);
+ }
+
+ /*
+ * Did we write the wrong # entries in the header?
+ * If so, rewrite it like in fast-import
+ */
+ if (pack_to_stdout || nr_written == nr_remaining) {
+ sha1close(f, pack_file_sha1, 1);
+ } else {
+ sha1close(f, pack_file_sha1, 0);
+ fixup_header_footer(f->fd, pack_file_sha1, pack_tmp_name, nr_written);
+ }
+
+ if (!pack_to_stdout) {
+ unsigned char object_list_sha1[20];
+ mode_t mode = umask(0);
+
+ umask(mode);
+ mode = 0444 & ~mode;
+
+ write_index_file(last_obj_offset, object_list_sha1);
+ snprintf(tmpname, sizeof(tmpname), "%s-%s.pack",
+ base_name, sha1_to_hex(object_list_sha1));
+ if (adjust_perm(pack_tmp_name, mode))
+ die("unable to make temporary pack file readable: %s",
+ strerror(errno));
+ if (rename(pack_tmp_name, tmpname))
+ die("unable to rename temporary pack file: %s",
+ strerror(errno));
+ snprintf(tmpname, sizeof(tmpname), "%s-%s.idx",
+ base_name, sha1_to_hex(object_list_sha1));
+ if (adjust_perm(idx_tmp_name, mode))
+ die("unable to make temporary index file readable: %s",
+ strerror(errno));
+ if (rename(idx_tmp_name, tmpname))
+ die("unable to rename temporary index file: %s",
+ strerror(errno));
+ puts(sha1_to_hex(object_list_sha1));
+ }
+
+ /* mark written objects as written to previous pack */
+ for (j = 0; j < nr_written; j++) {
+ written_list[j]->offset = (off_t)-1;
+ }
+ nr_remaining -= nr_written;
+ } while (nr_remaining && i < nr_objects);
+
+ free(written_list);
if (do_progress)
stop_progress(&progress_state);
- done:
if (written != nr_result)
die("wrote %u objects while expecting %u", written, nr_result);
- sha1close(f, pack_file_sha1, 1);
-
- if (!pack_to_stdout) {
- unsigned char object_list_sha1[20];
- mode_t mode = umask(0);
-
- umask(mode);
- mode = 0444 & ~mode;
-
- write_index_file(last_obj_offset, object_list_sha1);
- snprintf(tmpname, sizeof(tmpname), "%s-%s.pack",
- base_name, sha1_to_hex(object_list_sha1));
- if (adjust_perm(pack_tmp_name, mode))
- die("unable to make temporary pack file readable: %s",
- strerror(errno));
- if (rename(pack_tmp_name, tmpname))
- die("unable to rename temporary pack file: %s",
- strerror(errno));
- snprintf(tmpname, sizeof(tmpname), "%s-%s.idx",
- base_name, sha1_to_hex(object_list_sha1));
- if (adjust_perm(idx_tmp_name, mode))
- die("unable to make temporary index file readable: %s",
- strerror(errno));
- if (rename(idx_tmp_name, tmpname))
- die("unable to rename temporary index file: %s",
- strerror(errno));
- puts(sha1_to_hex(object_list_sha1));
+ for (j = 0; i < nr_objects; i++) {
+ struct object_entry *e = objects + i;
+ j += !e->offset && !e->preferred_base;
}
+ if (j)
+ die("wrote %u objects as expected but %u unwritten", written, j);
}
static int sha1_sort(const void *_a, const void *_b)
@@ -746,18 +775,11 @@ static void write_index_file(off_t last_obj_offset, unsigned char *sha1)
idx_tmp_name = xstrdup(tmpname);
f = sha1fd(fd, idx_tmp_name);
- if (nr_result) {
- uint32_t j = 0;
- sorted_by_sha =
- xcalloc(nr_result, sizeof(struct object_entry *));
- for (i = 0; i < nr_objects; i++)
- if (!objects[i].preferred_base)
- sorted_by_sha[j++] = objects + i;
- if (j != nr_result)
- die("listed %u objects while expecting %u", j, nr_result);
- qsort(sorted_by_sha, nr_result, sizeof(*sorted_by_sha), sha1_sort);
+ if (nr_written) {
+ sorted_by_sha = written_list;
+ qsort(sorted_by_sha, nr_written, sizeof(*sorted_by_sha), sha1_sort);
list = sorted_by_sha;
- last = sorted_by_sha + nr_result;
+ last = sorted_by_sha + nr_written;
} else
sorted_by_sha = list = last = NULL;
@@ -795,7 +817,7 @@ static void write_index_file(off_t last_obj_offset, unsigned char *sha1)
/* Write the actual SHA1 entries. */
list = sorted_by_sha;
- for (i = 0; i < nr_result; i++) {
+ for (i = 0; i < nr_written; i++) {
struct object_entry *entry = *list++;
if (index_version < 2) {
uint32_t offset = htonl(entry->offset);
@@ -810,7 +832,7 @@ static void write_index_file(off_t last_obj_offset, unsigned char *sha1)
/* write the crc32 table */
list = sorted_by_sha;
- for (i = 0; i < nr_objects; i++) {
+ for (i = 0; i < nr_written; i++) {
struct object_entry *entry = *list++;
uint32_t crc32_val = htonl(entry->crc32);
sha1write(f, &crc32_val, 4);
@@ -818,7 +840,7 @@ static void write_index_file(off_t last_obj_offset, unsigned char *sha1)
/* write the 32-bit offset table */
list = sorted_by_sha;
- for (i = 0; i < nr_objects; i++) {
+ for (i = 0; i < nr_written; i++) {
struct object_entry *entry = *list++;
uint32_t offset = (entry->offset <= index_off32_limit) ?
entry->offset : (0x80000000 | nr_large_offset++);
@@ -843,7 +865,6 @@ static void write_index_file(off_t last_obj_offset, unsigned char *sha1)
sha1write(f, pack_file_sha1, 20);
sha1close(f, NULL, 1);
- free(sorted_by_sha);
SHA1_Final(sha1, &ctx);
}
--
1.5.2.rc0.789.gd951
^ permalink raw reply related
* [PATCH 5/5] git-repack --max-pack-size: add option parsing to enable feature
From: Dana How @ 2007-05-02 0:08 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Git Mailing List, danahow
Add --max-pack-size parsing and usage messages.
Upgrade git-repack.sh to handle multiple packfile names.
Signed-off-by: Dana L. How <danahow@gmail.com>
---
builtin-pack-objects.c | 9 ++++++++-
git-repack.sh | 12 +++++++-----
2 files changed, 15 insertions(+), 6 deletions(-)
diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index a7b19c2..69fec34 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -15,7 +15,7 @@
#include "progress.h"
static const char pack_usage[] = "\
-git-pack-objects [{ -q | --progress | --all-progress }] \n\
+git-pack-objects [{ -q | --progress | --all-progress }] [--max-pack-size=N] \n\
[--local] [--incremental] [--window=N] [--depth=N] \n\
[--no-reuse-delta] [--delta-base-offset] [--non-empty] \n\
[--revs [--unpacked | --all]*] [--reflog] [--stdout | base-name] \n\
@@ -1752,6 +1752,13 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
incremental = 1;
continue;
}
+ if (!prefixcmp(arg, "--max-pack-size=")) {
+ char *end;
+ pack_size_limit = strtoul(arg+16, &end, 0) * 1024 * 1024;
+ if (!arg[16] || *end)
+ usage(pack_usage);
+ continue;
+ }
if (!prefixcmp(arg, "--window=")) {
char *end;
window = strtoul(arg+9, &end, 0);
diff --git a/git-repack.sh b/git-repack.sh
index ddfa8b4..ebce1b3 100755
--- a/git-repack.sh
+++ b/git-repack.sh
@@ -3,7 +3,7 @@
# Copyright (c) 2005 Linus Torvalds
#
-USAGE='[-a] [-d] [-f] [-l] [-n] [-q] [--window=N] [--depth=N]'
+USAGE='[-a] [-d] [-f] [-l] [-n] [-q] [--max-pack-size=N] [--window=N] [--depth=N]'
SUBDIRECTORY_OK='Yes'
. git-sh-setup
@@ -18,6 +18,7 @@ do
-q) quiet=-q ;;
-f) no_reuse_delta=--no-reuse-delta ;;
-l) local=--local ;;
+ --max-pack-size=*) extra="$extra $1" ;;
--window=*) extra="$extra $1" ;;
--depth=*) extra="$extra $1" ;;
*) usage ;;
@@ -62,11 +63,12 @@ case ",$all_into_one," in
esac
args="$args $local $quiet $no_reuse_delta$extra"
-name=$(git-pack-objects --non-empty --all --reflog $args </dev/null "$PACKTMP") ||
+names=$(git-pack-objects --non-empty --all --reflog $args </dev/null "$PACKTMP") ||
exit 1
-if [ -z "$name" ]; then
+if [ -z "$names" ]; then
echo Nothing new to pack.
-else
+fi
+for name in $names ; do
chmod a-w "$PACKTMP-$name.pack"
chmod a-w "$PACKTMP-$name.idx"
if test "$quiet" != '-q'; then
@@ -92,7 +94,7 @@ else
exit 1
}
rm -f "$PACKDIR/old-pack-$name.pack" "$PACKDIR/old-pack-$name.idx"
-fi
+done
if test "$remove_redundant" = t
then
--
1.5.2.rc0.789.gd951
^ permalink raw reply related
* [PATCH] cvsserver: Handle re-added files correctly
From: Frank Lichtenheld @ 2007-05-02 0:43 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Martin Langhoff, Frank Lichtenheld
We can't unconditionally assign revision 1.1 to
newly added files. In case the file did exist in the
past and was deleted we need to honor the old
revision number.
Signed-off-by: Frank Lichtenheld <frank@lichtenheld.de>
---
git-cvsserver.perl | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
Found due to working on the test cases.
diff --git a/git-cvsserver.perl b/git-cvsserver.perl
index 087e3ab..3e7bf5b 100755
--- a/git-cvsserver.perl
+++ b/git-cvsserver.perl
@@ -2519,7 +2519,7 @@ sub update
#$log->debug("ADDED $name");
$head->{$name} = {
name => $name,
- revision => 1,
+ revision => $head->{$name}{revision} ? $head->{$name}{revision}+1 : 1,
filehash => $hash,
commithash => $commit->{hash},
modified => $commit->{date},
--
1.5.1.2
^ permalink raw reply related
* [PATCH] cvsserver: Add test cases for git-cvsserver
From: Frank Lichtenheld @ 2007-05-02 0:45 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Martin Langhoff, Frank Lichtenheld
In-Reply-To: <7vfy6h7wsb.fsf@assigned-by-dhcp.cox.net>
Use the :fork: access method to force cvs to
call "$CVS_SERVER server" even when accessing a local
repository.
Add a basic test for checkout and some tests for update.
Signed-off-by: Frank Lichtenheld <frank@lichtenheld.de>
---
t/t9400-git-cvsserver-server.sh | 120 +++++++++++++++++++++++++++++++++++++++
1 files changed, 120 insertions(+), 0 deletions(-)
create mode 100644 t/t9400-git-cvsserver-server.sh
Still my first test cases so careful review and comments
on style and implementation would be very welcome :)
diff --git a/t/t9400-git-cvsserver-server.sh b/t/t9400-git-cvsserver-server.sh
new file mode 100644
index 0000000..19b504b
--- /dev/null
+++ b/t/t9400-git-cvsserver-server.sh
@@ -0,0 +1,120 @@
+#!/bin/sh
+#
+# Copyright (c) 2007 Frank Lichtenheld
+#
+
+test_description='git-cvsserver access
+
+tests read access to a git repository with the
+cvs CLI client via git-cvsserver server'
+
+. ./test-lib.sh
+
+cvs >/dev/null 2>&1
+if test $? -ne 1
+then
+ test_expect_success 'skipping git-cvsserver tests, cvs not found' :
+ test_done
+ exit
+fi
+
+unset GIT_DIR GIT_CONFIG
+WORKDIR=$(pwd)
+SERVERDIR=$(pwd)/gitcvs.git
+CVSROOT=":fork:$SERVERDIR"
+CVSWORK=$(pwd)/cvswork
+CVS_SERVER=git-cvsserver
+export CVSROOT CVS_SERVER
+
+rm -rf "$CVSWORK" "$SERVERDIR"
+echo >empty &&
+ git add empty &&
+ git commit -q -m "First Commit" &&
+ git clone -q --local --bare "$WORKDIR/.git" "$SERVERDIR" >/dev/null 2>&1 &&
+ GIT_DIR="$SERVERDIR" git config --bool gitcvs.enabled true &&
+ GIT_DIR="$SERVERDIR" git config --bool gitcvs.logfile "$SERVERDIR/gitcvs.log" ||
+ exit 1
+
+# note that cvs doesn't accept absolute pathnames
+# as argument to co -d
+test_expect_success 'basic checkout' \
+ 'cvs -Q co -d cvswork master &&
+ test "$(echo $(grep -v ^D cvswork/CVS/Entries|cut -d/ -f2,3,5))" = "empty/1.1/"'
+
+test_expect_success 'cvs update (create new file)' \
+ 'echo testfile1 >testfile1 &&
+ git add testfile1 &&
+ git commit -q -m "Add testfile1" &&
+ git push gitcvs.git >/dev/null &&
+ cd cvswork &&
+ cvs -Q update &&
+ test "$(echo $(grep testfile1 CVS/Entries|cut -d/ -f2,3,5))" = "testfile1/1.1/" &&
+ diff -q testfile1 ../testfile1'
+
+cd "$WORKDIR"
+test_expect_success 'cvs update (update existing file)' \
+ 'echo line 2 >>testfile1 &&
+ git add testfile1 &&
+ git commit -q -m "Append to testfile1" &&
+ git push gitcvs.git >/dev/null &&
+ cd cvswork &&
+ cvs -Q update &&
+ test "$(echo $(grep testfile1 CVS/Entries|cut -d/ -f2,3,5))" = "testfile1/1.2/" &&
+ diff -q testfile1 ../testfile1'
+
+cd "$WORKDIR"
+#TODO: cvsserver doesn't support update w/o -d
+test_expect_failure "cvs update w/o -d doesn't create subdir (TODO)" \
+ 'mkdir test &&
+ echo >test/empty &&
+ git add test &&
+ git commit -q -m "Single Subdirectory" &&
+ git push gitcvs.git >/dev/null &&
+ cd cvswork &&
+ cvs -Q update &&
+ test ! -d test'
+
+cd "$WORKDIR"
+test_expect_success 'cvs update (subdirectories)' \
+ '(for dir in A A/B A/B/C A/D E; do
+ mkdir $dir &&
+ echo "test file in $dir" >"$dir/file_in_$(echo $dir|sed -e "s#/# #g")" &&
+ git add $dir;
+ done) &&
+ git commit -q -m "deep sub directory structure" &&
+ git push gitcvs.git >/dev/null &&
+ cd cvswork &&
+ cvs -Q update -d &&
+ (for dir in A A/B A/B/C A/D E; do
+ filename="file_in_$(echo $dir|sed -e "s#/# #g")" &&
+ if test "$(echo $(grep -v ^D $dir/CVS/Entries|cut -d/ -f2,3,5))" = "$filename/1.1/" &&
+ diff -q "$dir/$filename" "../$dir/$filename"; then
+ :
+ else
+ echo >failure
+ fi
+ done) &&
+ test ! -f failure'
+
+cd "$WORKDIR"
+test_expect_success 'cvs update (delete file)' \
+ 'git rm testfile1 &&
+ git commit -q -m "Remove testfile1" &&
+ git push gitcvs.git >/dev/null &&
+ cd cvswork &&
+ cvs -Q update &&
+ test -z "$(grep testfile1 CVS/Entries)" &&
+ test ! -f testfile1'
+
+cd "$WORKDIR"
+test_expect_success 'cvs update (re-add deleted file)' \
+ 'echo readded testfile >testfile1 &&
+ git add testfile1 &&
+ git commit -q -m "Re-Add testfile1" &&
+ git push gitcvs.git >/dev/null &&
+ cd cvswork &&
+ cvs -Q update &&
+ test "$(echo $(grep testfile1 CVS/Entries|cut -d/ -f2,3,5))" = "testfile1/1.4/" &&
+ diff -q testfile1 ../testfile1'
+
+test_done
--
1.5.1.2
^ permalink raw reply related
* [RFD/PATCH] Implement pack.compression and pack-objects --compression=N
From: Dana How @ 2007-05-02 3:18 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Git Mailing List, danahow
Git's object store and packing are optimized for
* Expensive repo to repo bandwidth; and
* Small-ish files
This justifies extensive use of compression.
In a multi-developer *office* with inter-repository
transfers occurring over a 100Mb+ LAN, there is less
reason to compress files and slow down response times.
Response times suffer even more when large files are involved.
However, *off-line* pack compression may still be
desirable to reduce storage space.
Consequently, for such a usage pattern it is useful
to specify different compression levels for loose
objects and packs. This patch implements a config
variable pack.compression in addition to the existing
core.compression, meant to be used for repacking.
It also adds --compression=N to pack-objects,
meant for push/pull/fetch, if different, or if different
on a per-repository basis.
** THIS PATCH IS UNTESTED AND MEANT FOR DISCUSSION. **
git-repack.sh might also need to be modified,
and how to pass --compression=N during push/pull/fetch
has not been investigated.
This applies on top of the git-repack --max-pack-size patchset.
Signed-off-by: Dana L. How <danahow@gmail.com>
---
builtin-pack-objects.c | 56 ++++++++++++++++++++++++++++++++++++------------
1 files changed, 42 insertions(+), 14 deletions(-)
diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index 69fec34..b663c15 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -70,6 +70,7 @@ static uint32_t pack_size_limit;
static int pack_to_stdout;
static int num_preferred_base;
static struct progress progress_state;
+static int pack_compression_level, pack_compression_seen;
/*
* The object names in objects array are hashed with this hashtable,
@@ -414,6 +415,16 @@ static unsigned long write_object(struct sha1file *f,
/* write limit if limited packsize and not first object */
unsigned long limit = pack_size_limit && nr_written ?
pack_size_limit - write_offset : 0;
+ /* no if no delta */
+ int usable_delta = !entry->delta ? 0 :
+ /* yes if unlimited packfile */
+ !pack_size_limit ? 1 :
+ /* no if base written to previous pack */
+ entry->delta->offset == (off_t)-1 ? 0 :
+ /* otherwise double-check written to this
+ * pack, like we do below
+ */
+ entry->delta->offset ? 1 : 0;
if (!pack_to_stdout)
crc32_begin(f);
@@ -423,8 +434,7 @@ static unsigned long write_object(struct sha1file *f,
to_reuse = 0; /* can't reuse what we don't have */
else if (obj_type == OBJ_REF_DELTA || obj_type == OBJ_OFS_DELTA)
/* check_object() decided it for us ... */
- to_reuse = !pack_size_limit ||
- (entry->delta->offset && entry->delta->offset != (off_t)-1);
+ to_reuse = usable_delta;
/* ... but pack split may override that */
else if (obj_type != entry->in_pack_type)
to_reuse = 0; /* pack has delta which is unusable */
@@ -435,6 +445,10 @@ static unsigned long write_object(struct sha1file *f,
* and we do not need to deltify it.
*/
+ /* differing core & pack compression when loose object -> must recompress */
+ if (!entry->in_pack && pack_compression_level != zlib_compression_level)
+ to_reuse = 0;
+ else
if (!entry->in_pack && !entry->delta) {
unsigned char *map;
unsigned long mapsize;
@@ -462,16 +476,6 @@ static unsigned long write_object(struct sha1file *f,
z_stream stream;
unsigned long maxsize;
void *out;
- /* no if no delta */
- int usable_delta = !entry->delta ? 0 :
- /* yes if unlimited packfile */
- !pack_size_limit ? 1 :
- /* no if base written to previous pack */
- entry->delta->offset == (off_t)-1 ? 0 :
- /* otherwise double-check written to this
- * pack, like we do below
- */
- entry->delta->offset ? 1 : 0;
buf = read_sha1_file(entry->sha1, &type, &size);
if (!buf)
die("unable to read %s", sha1_to_hex(entry->sha1));
@@ -493,7 +497,7 @@ static unsigned long write_object(struct sha1file *f,
}
/* compress the data to store and put compressed length in datalen */
memset(&stream, 0, sizeof(stream));
- deflateInit(&stream, zlib_compression_level);
+ deflateInit(&stream, pack_compression_level);
maxsize = deflateBound(&stream, size);
out = xmalloc(maxsize);
/* Compress it */
@@ -606,7 +610,7 @@ static unsigned long write_object(struct sha1file *f,
unuse_pack(&w_curs);
reused++;
}
- if (entry->delta)
+ if (usable_delta)
written_delta++;
written++;
if (!pack_to_stdout)
@@ -1622,6 +1626,16 @@ static int git_pack_config(const char *k, const char *v)
window = git_config_int(k, v);
return 0;
}
+ if (!strcmp(k, "pack.compression")) {
+ int level = git_config_int(k, v);
+ if (level == -1)
+ level = Z_DEFAULT_COMPRESSION;
+ else if (level < 0 || level > Z_BEST_COMPRESSION)
+ die("bad pack compression level %d", level);
+ pack_compression_level = level;
+ pack_compression_seen = 1;
+ return 0;
+ }
return git_default_config(k, v);
}
@@ -1732,6 +1746,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
rp_ac = 2;
git_config(git_pack_config);
+ if (!pack_compression_seen)
+ pack_compression_level = zlib_compression_level;
progress = isatty(2);
for (i = 1; i < argc; i++) {
@@ -1759,6 +1775,18 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
usage(pack_usage);
continue;
}
+ if (!prefixcmp(arg, "--compression=")) {
+ char *end;
+ int level = strtoul(arg+14, &end, 0);
+ if (!arg[14] || *end)
+ usage(pack_usage);
+ if (level == -1)
+ level = Z_DEFAULT_COMPRESSION;
+ else if (level < 0 || level > Z_BEST_COMPRESSION)
+ die("bad pack compression level %d", level);
+ pack_compression_level = level;
+ continue;
+ }
if (!prefixcmp(arg, "--window=")) {
char *end;
window = strtoul(arg+9, &end, 0);
--
1.5.2.rc0.787.g0014
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox