* Re: [PATCH] gitk: UTF-8 support
From: Pavel Roskin @ 2005-11-24 4:53 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Paul Mackerras, git
In-Reply-To: <7v64qi50sw.fsf@assigned-by-dhcp.cox.net>
Hello!
Quoting Junio C Hamano <junkio@cox.net>:
> Pavel Roskin <proski@gnu.org> writes:
>
> > Add gitencoding variable and set it to "utf-8". Use it for converting
> > git-rev-list output.
>
> Sounds good, but is it necessary? Unless I am grossly mistaken,
> I am opposed to this patch.
>
> When I run gitk with LANG and/or LC_CTYPE set to ja_JP.utf8 (I
> suspect *whatever*.utf8 would work the same way) on git.git
> repository, I see Lukas's name (originally in iso8859-1 but my
> commit objects have it in utf8) and Yoshifuji-san's name
> (iso2022 converted to utf8) just fine.
I see. I always use C locale.
> And when I run gitk with LANG and/or LC_CTYPE set to ja_JP.ujis
> (that is another name for EUC-JP) on a toy repository I have
> commit log messages in EUC-JP (I am not recommending that, just
> pointing out a possibility), I can see them just fine. In that
> test repository, setting locale to *.utf8 would not work.
Then what would you do to work with a repository using utf-8 if the current
locale is not utf-8?
> So I suspect your change breaks projects that use local
> encodings, without fixing or adding anything new.
I'll be away from any sane OS until Monday, but I assume my patch should help
those whose locale is set to an encoding other than utf-8 if they want to use a
repository using utf-8.
Anyway, I see your point. Not ever git repository uses utf-8. It is not
enforced by git.
So we have two solutions. One is to enforce utf-8 locale in git and force all
"offenders" to convert. That's probably too intrusive for now.
The other solution is to have a publicly available file under .git that would
keep the encoding name for the metadata (user names, logs etc). gitk could use
that file now. git could implement conversions a bit later. Maybe git could
warn about mismatching encoding as the first step.
--
Regards,
Pavel Roskin
^ permalink raw reply
* Re: gitweb on kernel.org and UTF-8
From: Ryan Anderson @ 2005-11-24 5:01 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Kay Sievers, Paul Mackerras, git
In-Reply-To: <7vfypm20eh.fsf@assigned-by-dhcp.cox.net>
On Wed, Nov 23, 2005 at 07:24:38PM -0800, Junio C Hamano wrote:
>
> How about doing something like this?
>
> [i18n]
> commitEncoding = utf8
> blobEncoding = utf8
>
> to mean:
>
> If you _have_ to make an assumption on an encoding
> commit and blob objects are in, utf8 is your best bet
> (but mistakes can happen, and some blobs can be binary).
The rest of the options help clarify this, but can you make these
options 'assumeCommitEncoding' and 'assumeBlobEncoding' to make it clear
that these are *assumptions* and not actually controlling what gets
written?
--
Ryan Anderson
sometimes Pug Majere
^ permalink raw reply
* Re: [RFC/PATCH 1/3] git-find-git: a new helper.
From: Marco Costalba @ 2005-11-24 5:41 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, Martin Atukunda
In-Reply-To: <7vmzjuzphl.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano wrote:
> git-find-git uses it to find the
> top-level, and if it was run from a subdirectory, outputs shell
> script fragments that can be evaled to define the two variables:
>
> GIT_DIR_PREFIX is a string to be prepended to cwd relative
> paths to make them repository relative.
>
> GIT_DIR_UPLEVEL is a string (e.g. "../") for the command to
> use to cd to the top-level of repository.
>
>
Sounds very good.
I did't had the time to test it but I think I can now
easily update qgit to run from a subdirectory with this new helper.
Marco
___________________________________
Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB
http://mail.yahoo.it
^ permalink raw reply
* Re: gitweb on kernel.org and UTF-8
From: Junio C Hamano @ 2005-11-24 6:19 UTC (permalink / raw)
To: Ryan Anderson; +Cc: git
In-Reply-To: <20051124050104.GC16995@mythryan2.michonline.com>
Ryan Anderson <ryan@michonline.com> writes:
> On Wed, Nov 23, 2005 at 07:24:38PM -0800, Junio C Hamano wrote:
>>
>> How about doing something like this?
>>
>> [i18n]
>> commitEncoding = utf8
>> blobEncoding = utf8
>>
>> to mean:
>>
>> If you _have_ to make an assumption on an encoding
>> commit and blob objects are in, utf8 is your best bet
>> (but mistakes can happen, and some blobs can be binary).
>
> The rest of the options help clarify this, but can you make these
> options 'assumeCommitEncoding' and 'assumeBlobEncoding' to make it clear
> that these are *assumptions* and not actually controlling what gets
> written?
As I outlined in the "editorEncoding" part, if everything works
as planned, your latin-1 editing editor would leave latin-1
message for git-commit to pick up (or command line "-m $msg"
option would be encoded in latin-1), and iconv would munge that
to utf8 to feed commit-tree (because of "commitEncoding" being
utf8). In that sense, commitEncoding is not assumption for the
writers. If everybody, including outside sources we merge from,
makes best effort not to screw up, these settings would
faithfully describe what encoding logs are in.
But writers can screw up, and funnily encoded commit messages
merge from outside source brings in cannot be fixed after the
fact, so "assume" part must be implied anyway for readers.
^ permalink raw reply
* Re: [RFC/PATCH 1/3] git-find-git: a new helper.
From: Junio C Hamano @ 2005-11-24 6:21 UTC (permalink / raw)
To: Marco Costalba; +Cc: git
In-Reply-To: <4385528D.7010203@yahoo.it>
Marco Costalba <mcostalba@yahoo.it> writes:
> Junio C Hamano wrote:
>
>> git-find-git uses it to find the
>> top-level, and if it was run from a subdirectory, outputs shell
>> script fragments that can be evaled to define the two variables:
>> GIT_DIR_PREFIX is a string to be prepended to cwd relative
>> paths to make them repository relative.
>> GIT_DIR_UPLEVEL is a string (e.g. "../") for the command to
>> use to cd to the top-level of repository.
>>
> Sounds very good.
>
> I did't had the time to test it but I think I can now
> easily update qgit to run from a subdirectory with this new helper.
The helper might be good, but it turns out that the idea of
using it from git-sh-setup was not that great, so please
disregard the third part of the patch. I think I need a bit
more thinking about this.
^ permalink raw reply
* Re: [PATCH] gitk: UTF-8 support
From: Junio C Hamano @ 2005-11-24 6:23 UTC (permalink / raw)
To: Pavel Roskin; +Cc: Paul Mackerras, git
In-Reply-To: <20051123235315.oht0zjidc4ccg8gs@webmail.spamcop.net>
Pavel Roskin <proski@gnu.org> writes:
> Then what would you do to work with a repository using utf-8 if the current
> locale is not utf-8?
Obviously the same way as I did to try things out:
$ LANG=en_US.utf8 DISPLAY=:0 gitk blah
> Anyway, I see your point. Not ever git repository uses utf-8. It is not
> enforced by git.
That is not the point. Point is that I think the user can use
LANG and LC_ALL (I suspect LC_CTYPE is what matters) to get what
you want, and I suspect hardcoding utf8 robs users the
possibility to deal with a repository that uses something else.
And as I suggested in another message (in the died-out thread
about gitweb), we could have i18n.commitEncoding in the
configuration to help gitk and gitweb. I think that is the same
as your "other option".
^ permalink raw reply
* Re: [PATCH] gitk: UTF-8 support
From: Pavel Roskin @ 2005-11-24 7:12 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Paul Mackerras, git
In-Reply-To: <7v1x16y36r.fsf@assigned-by-dhcp.cox.net>
Quoting Junio C Hamano <junkio@cox.net>:
> That is not the point. Point is that I think the user can use
> LANG and LC_ALL (I suspect LC_CTYPE is what matters) to get what
> you want, and I suspect hardcoding utf8 robs users the
> possibility to deal with a repository that uses something else.
Not to argue with you, but it's worth pointing out that git is heavily multiuser
software, and interoperability should not be ranked below local configurability.
> And as I suggested in another message (in the died-out thread
> about gitweb), we could have i18n.commitEncoding in the
> configuration to help gitk and gitweb. I think that is the same
> as your "other option".
Yes. Then my patch needs to be changed to set encoding to that setting and only
if it's present.
--
Regards,
Pavel Roskin
^ permalink raw reply
* Re: [PATCH] archimport improvements
From: Eric Wong @ 2005-11-24 7:46 UTC (permalink / raw)
To: Martin Langhoff; +Cc: git list
In-Reply-To: <46a038f90511170126l5f04f4e0ta578fc2cb375db68@mail.gmail.com>
Martin Langhoff <martin.langhoff@gmail.com> wrote:
> Eric,
>
> I've merged and pushed out to
> http://locke.catalyst.net.nz/git/git-martinlanghoff.git/#tojunio
>
> [PATCH 1/5] remove shellquote usage for tags
> [PATCH 2/5] archimport: don't die on merge-base failure
> [PATCH 3/5] Disambiguate the term 'branch' in Arch vs git
> [PATCH 1/2] archimport: allow for old style branch and public tag names
>
> That last one had a small edit to rebase it to the top of the head --
> will probably have a small conflict for you on the usage line and
> getopts() line.
>
> What is pending is...
>
> * [PATCH 4/5] Overhaul of changeset application
>
> I am testing it right now. Finding it rather slow on an idle linux
> workstation with fast IDE disks, no X.org loaded and 1GB or RAM.
> iowait is pegged at 90%. Wonder what will happen on a system with slow
> disk access. tla/baz are unusable under any OS where the fs stack is
> not _that_ polished (OSX and friends).
Ok, I didn't expect you guys to have 12k of files in your trees. None
of your source trees are remotely close to that size (but I have many
more changesets). I'm surprised you guys were able to put up
with Arch in the first place!
125m58.431s with my method.
8m24.504s with yours :)
All of my usual source trees imported 1k changesets in 10-15 minutes
> The early versions of the import also used $TLA for all ops, and I was
> forced to change it to get my repos transformed in a reasonable time.
>
> Can you send me a patch that makes it optional, so users can choose
> fast or correct? I don't want to force glacial imports on anyone,
> specially me. Testing an import of a reasonably sized repo must be a
> quick operation or I won't do it ;-) And I do work on OSX too.
Patches on the way.
OTOH, the time spent importing the bulk of the history is a one-time
operation for most people and I'd much rather it get things as right as
possible and move on.
> On the other hand, I might just implement renamed directories tracking
> separately, specially if someone can point me to a public repo with
> some interesting cases of renamed directories.
IIRC, there are several nasty cases all of which are ordering-related,
especially with regard to nested directories or file renames inside
directories that are also renamed. It should be noted that not even tla
gets all the possible directory rename cases right (baz seems better
from my observations).
> These patches seem to hang from 4/5 so will need rebasing after a
> reworked. The first one seems to be 3 or 4 patches in one. It'd be
> good to break it up.
Sorry, I rushed through the initial overhaul and didn't generate neat
patches because I wanted to get some of my work moved to git ASAP.
> * [ PATCH 5/5] -D <depth> option to recurse into merged branches
> * Re: [PATCH 5/5] -D <depth> option to recurse into merged branches
> * [PATCH 2/2] archimport: sync_to_ps() messages for tracking tla methods
--
Eric Wong
^ permalink raw reply
* [PATCH 1/9] archimport: first, make sure it still compiles
From: Eric Wong @ 2005-11-24 7:47 UTC (permalink / raw)
To: Martin Langhoff; +Cc: git list, Martin Langhoff
In-Reply-To: <20051124074605.GA4789@mail.yhbt.net>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
git-archimport.perl | 16 ++++++++++++++++
1 files changed, 16 insertions(+), 0 deletions(-)
applies-to: a17c1f442587b9c7d68b4f7e08c5f6786599c61e
119b07aa2bdb23d5f4977c4d696dd5e7eea56ca6
diff --git a/git-archimport.perl b/git-archimport.perl
index c3bed08..b5f8a2c 100755
--- a/git-archimport.perl
+++ b/git-archimport.perl
@@ -99,6 +99,7 @@ my %psets = (); # the co
my %rptags = (); # my reverse private tags
# to map a SHA1 to a commitid
+my $TLA = $ENV{'ARCH_CLIENT'} || 'tla';
foreach my $root (@arch_roots) {
my ($arepo, $abranch) = split(m!/!, $root);
@@ -850,3 +851,18 @@ sub commitid2pset {
|| (print Dumper(sort keys %psets)) && die "Cannot find patchset for $name";
return $ps;
}
+
+# an alterative to `command` that allows input to be passed as an array
+# to work around shell problems with weird characters in arguments
+sub safe_pipe_capture {
+ my @output;
+ if (my $pid = open my $child, '-|') {
+ @output = (<$child>);
+ close $child or die join(' ',@_).": $! $?";
+ } else {
+ exec(@_) or die $?; # exec() can fail the executable can't be found
+ }
+ return wantarray ? @output : join('',@output);
+}
+
+
---
0.99.9.GIT
--
Eric Wong
^ permalink raw reply related
* [PATCH 2/9] remove String::ShellQuote dependency.
From: Eric Wong @ 2005-11-24 7:48 UTC (permalink / raw)
To: Martin Langhoff; +Cc: git list, Martin Langhoff
In-Reply-To: <20051124074739.GB4789@mail.yhbt.net>
use safe_pipe_capture() or system() over backticks where
shellquoting may have been necessary.
More changes planned, so I'm not touching the parts I'm
planning on replacing entirely.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
git-archimport.perl | 51 ++++++++++++++++++++++++++++-----------------------
1 files changed, 28 insertions(+), 23 deletions(-)
applies-to: 83307766d30e928179b9aa85a3d7bb906cc08846
80494a7d496ab9f6e0a76a60b1f0b4215fdff442
diff --git a/git-archimport.perl b/git-archimport.perl
index b5f8a2c..b7e2480 100755
--- a/git-archimport.perl
+++ b/git-archimport.perl
@@ -55,9 +55,8 @@ use warnings;
use Getopt::Std;
use File::Spec;
use File::Temp qw(tempfile tempdir);
-use File::Path qw(mkpath);
+use File::Path qw(mkpath rmtree);
use File::Basename qw(basename dirname);
-use String::ShellQuote;
use Time::Local;
use IO::Socket;
use IO::Pipe;
@@ -306,7 +305,7 @@ foreach my $ps (@psets) {
unless ($import) { # skip for import
if ( -e "$git_dir/refs/heads/$ps->{branch}") {
# we know about this branch
- `git checkout $ps->{branch}`;
+ system('git-checkout',$ps->{branch});
} else {
# new branch! we need to verify a few things
die "Branch on a non-tag!" unless $ps->{type} eq 't';
@@ -315,7 +314,7 @@ foreach my $ps (@psets) {
unless $branchpoint;
# find where we are supposed to branch from
- `git checkout -b $ps->{branch} $branchpoint`;
+ system('git-checkout','-b',$ps->{branch},$branchpoint);
# If we trust Arch with the fact that this is just
# a tag, and it does not affect the state of the tree
@@ -344,7 +343,7 @@ foreach my $ps (@psets) {
#
my $tree;
- my $commitlog = `tla cat-archive-log -A $ps->{repo} $ps->{id}`;
+ my $commitlog = safe_pipe_capture($TLA,'cat-archive-log',$ps->{id});
die "Error in cat-archive-log: $!" if $?;
# parselog will git-add/rm files
@@ -422,7 +421,7 @@ foreach my $ps (@psets) {
#
my @par;
if ( -e "$git_dir/refs/heads/$ps->{branch}") {
- if (open HEAD, "<$git_dir/refs/heads/$ps->{branch}") {
+ if (open HEAD, "<","$git_dir/refs/heads/$ps->{branch}") {
my $p = <HEAD>;
close HEAD;
chomp $p;
@@ -437,7 +436,6 @@ foreach my $ps (@psets) {
if ($ps->{merges}) {
push @par, find_parents($ps);
}
- my $par = join (' ', @par);
#
# Commit, tag and clean state
@@ -454,7 +452,7 @@ foreach my $ps (@psets) {
$commit_rh = 'commit_rh';
$commit_wh = 'commit_wh';
- $pid = open2(*READER, *WRITER, "git-commit-tree $tree $par")
+ $pid = open2(*READER, *WRITER,'git-commit-tree',$tree,@par)
or die $!;
print WRITER $logmessage; # write
close WRITER;
@@ -469,7 +467,7 @@ foreach my $ps (@psets) {
#
# Update the branch
#
- open HEAD, ">$git_dir/refs/heads/$ps->{branch}";
+ open HEAD, ">","$git_dir/refs/heads/$ps->{branch}";
print HEAD $commitid;
close HEAD;
system('git-update-ref', 'HEAD', "$ps->{branch}");
@@ -483,21 +481,23 @@ foreach my $ps (@psets) {
print " + tree $tree\n";
print " + commit $commitid\n";
$opt_v && print " + commit date is $ps->{date} \n";
- $opt_v && print " + parents: $par \n";
+ $opt_v && print " + parents: ",join(' ',@par),"\n";
}
sub apply_import {
my $ps = shift;
my $bname = git_branchname($ps->{id});
- `mkdir -p $tmp`;
+ mkpath($tmp);
- `tla get -s --no-pristine -A $ps->{repo} $ps->{id} $tmp/import`;
+ safe_pipe_capture($TLA,'get','-s','--no-pristine',$ps->{id},"$tmp/import");
die "Cannot get import: $!" if $?;
- `rsync -v --archive --delete --exclude '$git_dir' --exclude '.arch-ids' --exclude '{arch}' $tmp/import/* ./`;
+ system('rsync','-aI','--delete', '--exclude',$git_dir,
+ '--exclude','.arch-ids','--exclude','{arch}',
+ "$tmp/import/", './');
die "Cannot rsync import:$!" if $?;
- `rm -fr $tmp/import`;
+ rmtree("$tmp/import");
die "Cannot remove tempdir: $!" if $?;
@@ -507,10 +507,10 @@ sub apply_import {
sub apply_cset {
my $ps = shift;
- `mkdir -p $tmp`;
+ mkpath($tmp);
# get the changeset
- `tla get-changeset -A $ps->{repo} $ps->{id} $tmp/changeset`;
+ safe_pipe_capture($TLA,'get-changeset',$ps->{id},"$tmp/changeset");
die "Cannot get changeset: $!" if $?;
# apply patches
@@ -534,17 +534,20 @@ sub apply_cset {
$orig =~ s/\.modified$//; # lazy
$orig =~ s!^\Q$tmp\E/changeset/patches/!!;
#print "rsync -p '$mod' '$orig'";
- `rsync -p $mod ./$orig`;
+ system('rsync','-p',$mod,"./$orig");
die "Problem applying binary changes! $!" if $?;
}
}
# bring in new files
- `rsync --archive --exclude '$git_dir' --exclude '.arch-ids' --exclude '{arch}' $tmp/changeset/new-files-archive/* ./`;
+ system('rsync','-aI','--exclude',$git_dir,
+ '--exclude','.arch-ids',
+ '--exclude', '{arch}',
+ "$tmp/changeset/new-files-archive/",'./');
# deleted files are hinted from the commitlog processing
- `rm -fr $tmp/changeset`;
+ rmtree("$tmp/changeset");
}
@@ -622,9 +625,9 @@ sub parselog {
# tla cat-archive-log will give us filenames with spaces as file\(sp)name - why?
# we can assume that any filename with \ indicates some pika escaping that we want to get rid of.
if ($t =~ /\\/ ){
- $t = `tla escape --unescaped '$t'`;
+ $t = (safe_pipe_capture($TLA,'escape','--unescaped',$t))[0];
}
- push (@tmp, shell_quote($t));
+ push (@tmp, $t);
}
@$ref = @tmp;
}
@@ -827,8 +830,10 @@ sub find_parents {
}
}
}
- @parents = keys %parents;
- @parents = map { " -p " . ptag($_) } @parents;
+ @parents = ();
+ foreach (keys %parents) {
+ push @parents, '-p', ptag($_);
+ }
return @parents;
}
---
0.99.9.GIT
^ permalink raw reply related
* [PATCH 3/9] fix -t tmpdir switch
From: Eric Wong @ 2005-11-24 7:50 UTC (permalink / raw)
To: Martin Langhoff; +Cc: git list, Martin Langhoff
In-Reply-To: <20051124074857.GC4789@mail.yhbt.net>
set TMPDIR env correctly if -t <tmpdir> is passed from the command-line.
setting TMPDIR => 1 as an argument to tempdir() has no effect otherwise
Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
git-archimport.perl | 5 ++---
1 files changed, 2 insertions(+), 3 deletions(-)
applies-to: 015fcfae8cdd564f0993940c5bac303c41913b1a
25aadaf3ebc18fcc3c7948dc831d3f93447b03b6
diff --git a/git-archimport.perl b/git-archimport.perl
index b7e2480..2ed2e3c 100755
--- a/git-archimport.perl
+++ b/git-archimport.perl
@@ -88,9 +88,8 @@ usage if $opt_h;
@ARGV >= 1 or usage();
my @arch_roots = @ARGV;
-my ($tmpdir, $tmpdirname) = tempdir('git-archimport-XXXXXX', TMPDIR => 1, CLEANUP => 1);
-my $tmp = $opt_t || 1;
-$tmp = tempdir('git-archimport-XXXXXX', TMPDIR => 1, CLEANUP => 1);
+$ENV{'TMPDIR'} = $opt_t if $opt_t; # $ENV{TMPDIR} will affect tempdir() calls:
+my $tmp = tempdir('git-archimport-XXXXXX', TMPDIR => 1, CLEANUP => 1);
$opt_v && print "+ Using $tmp as temporary directory\n";
my @psets = (); # the collection
---
0.99.9.GIT
^ permalink raw reply related
* [PATCH 4/9] remove git wrapper dependency
From: Eric Wong @ 2005-11-24 7:51 UTC (permalink / raw)
To: Martin Langhoff; +Cc: git list, Martin Langhoff
In-Reply-To: <20051124075027.GD4789@mail.yhbt.net>
use git-diff-files instead of git diff-files so we don't rely on the
wrapper being installed (some people may have git as GNU interactive
tools :)
Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
git-archimport.perl | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
applies-to: 36c4ad6c2442e7700e5f1435d834f86d2680dd7f
40b5378433aa6b09dd358d482d7713c2db6a5d92
diff --git a/git-archimport.perl b/git-archimport.perl
index 2ed2e3c..938fa2b 100755
--- a/git-archimport.perl
+++ b/git-archimport.perl
@@ -278,7 +278,7 @@ foreach my $ps (@psets) {
#
# ensure we have a clean state
#
- if (`git diff-files`) {
+ if (`git-diff-files`) {
die "Unclean tree when about to process $ps->{id} " .
" - did we fail to commit cleanly before?";
}
---
0.99.9.GIT
^ permalink raw reply related
* [PATCH 5/9] add -D <depth> and -a switch
From: Eric Wong @ 2005-11-24 7:52 UTC (permalink / raw)
To: Martin Langhoff; +Cc: git list, Martin Langhoff
In-Reply-To: <20051124075133.GE4789@mail.yhbt.net>
add -D <depth> option to abrowse add -a switch to attempt to
auto-register archives at mirrors.sourcecontrol.net
Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
git-archimport.perl | 227 +++++++++++++++++++++++++++++++--------------------
1 files changed, 136 insertions(+), 91 deletions(-)
applies-to: 4fd19922030196431179453910cb2b850b54c17d
c52baf62ec29772f7a4e5b0aa2b605bf0f6aa8f7
diff --git a/git-archimport.perl b/git-archimport.perl
index 938fa2b..3968740 100755
--- a/git-archimport.perl
+++ b/git-archimport.perl
@@ -53,14 +53,9 @@ and can contain multiple, unrelated bran
use strict;
use warnings;
use Getopt::Std;
-use File::Spec;
-use File::Temp qw(tempfile tempdir);
+use File::Temp qw(tempdir);
use File::Path qw(mkpath rmtree);
use File::Basename qw(basename dirname);
-use Time::Local;
-use IO::Socket;
-use IO::Pipe;
-use POSIX qw(strftime dup2);
use Data::Dumper qw/ Dumper /;
use IPC::Open2;
@@ -71,27 +66,33 @@ my $git_dir = $ENV{"GIT_DIR"} || ".git";
$ENV{"GIT_DIR"} = $git_dir;
my $ptag_dir = "$git_dir/archimport/tags";
-our($opt_h,$opt_v, $opt_T,$opt_t,$opt_o);
+our($opt_h,$opt_v,$opt_T,$opt_t,$opt_D,$opt_a,$opt_o);
sub usage() {
print STDERR <<END;
Usage: ${\basename $0} # fetch/update GIT from Arch
- [ -o ] [ -h ] [ -v ] [ -T ] [ -t tempdir ]
+ [ -o ] [ -h ] [ -v ] [ -T ] [ -a ] [ -D depth ] [ -t tempdir ]
repository/arch-branch [ repository/arch-branch] ...
END
exit(1);
}
-getopts("Thvt:") or usage();
+getopts("Thvat:D:") or usage();
usage if $opt_h;
@ARGV >= 1 or usage();
-my @arch_roots = @ARGV;
+# $arch_branches:
+# values associated with keys:
+# =1 - Arch version / git 'branch' detected via abrowse on a limit
+# >1 - Arch version / git 'branch' of an auxilliary branch we've merged
+my %arch_branches = map { $_ => 1 } @ARGV;
$ENV{'TMPDIR'} = $opt_t if $opt_t; # $ENV{TMPDIR} will affect tempdir() calls:
my $tmp = tempdir('git-archimport-XXXXXX', TMPDIR => 1, CLEANUP => 1);
$opt_v && print "+ Using $tmp as temporary directory\n";
+my %reachable = (); # Arch repositories we can access
+my %unreachable = (); # Arch repositories we can't access :<
my @psets = (); # the collection
my %psets = (); # the collection, by name
@@ -99,96 +100,112 @@ my %rptags = (); # my rev
# to map a SHA1 to a commitid
my $TLA = $ENV{'ARCH_CLIENT'} || 'tla';
-foreach my $root (@arch_roots) {
- my ($arepo, $abranch) = split(m!/!, $root);
- open ABROWSE, "tla abrowse -f -A $arepo --desc --merges $abranch |"
- or die "Problems with tla abrowse: $!";
-
- my %ps = (); # the current one
- my $mode = '';
- my $lastseen = '';
-
- while (<ABROWSE>) {
- chomp;
+sub do_abrowse {
+ my $stage = shift;
+ while (my ($limit, $level) = each %arch_branches) {
+ next unless $level == $stage;
- # first record padded w 8 spaces
- if (s/^\s{8}\b//) {
-
- # store the record we just captured
- if (%ps) {
- my %temp = %ps; # break references
- push (@psets, \%temp);
- $psets{$temp{id}} = \%temp;
- %ps = ();
- }
-
- my ($id, $type) = split(m/\s{3}/, $_);
- $ps{id} = $id;
- $ps{repo} = $arepo;
-
- # deal with types
- if ($type =~ m/^\(simple changeset\)/) {
- $ps{type} = 's';
- } elsif ($type eq '(initial import)') {
- $ps{type} = 'i';
- } elsif ($type =~ m/^\(tag revision of (.+)\)/) {
- $ps{type} = 't';
- $ps{tag} = $1;
- } else {
- warn "Unknown type $type";
- }
- $lastseen = 'id';
- }
-
- if (s/^\s{10}//) {
- # 10 leading spaces or more
- # indicate commit metadata
+ open ABROWSE, "$TLA abrowse -fkD --merges $limit |"
+ or die "Problems with tla abrowse: $!";
+
+ my %ps = (); # the current one
+ my $lastseen = '';
+
+ while (<ABROWSE>) {
+ chomp;
- # date & author
- if ($lastseen eq 'id' && m/^\d{4}-\d{2}-\d{2}/) {
+ # first record padded w 8 spaces
+ if (s/^\s{8}\b//) {
+ my ($id, $type) = split(m/\s+/, $_, 2);
+
+ my %last_ps;
+ # store the record we just captured
+ if (%ps && !exists $psets{ $ps{id} }) {
+ %last_ps = %ps; # break references
+ push (@psets, \%last_ps);
+ $psets{ $last_ps{id} } = \%last_ps;
+ }
- my ($date, $authoremail) = split(m/\s{2,}/, $_);
- $ps{date} = $date;
- $ps{date} =~ s/\bGMT$//; # strip off trailign GMT
- if ($ps{date} =~ m/\b\w+$/) {
- warn 'Arch dates not in GMT?! - imported dates will be wrong';
+ my $branch = extract_versionname($id);
+ %ps = ( id => $id, branch => $branch );
+ if (%last_ps && ($last_ps{branch} eq $branch)) {
+ $ps{parent_id} = $last_ps{id};
+ }
+
+ $arch_branches{$branch} = 1;
+ $lastseen = 'id';
+
+ # deal with types (should work with baz or tla):
+ if ($type =~ m/\(.*changeset\)/) {
+ $ps{type} = 's';
+ } elsif ($type =~ /\(.*import\)/) {
+ $ps{type} = 'i';
+ } elsif ($type =~ m/\(tag.*\)/) {
+ $ps{type} = 't';
+ # read which revision we've tagged when we parse the log
+ #$ps{tag} = $1;
+ } else {
+ warn "Unknown type $type";
+ }
+
+ $arch_branches{$branch} = 1;
+ $lastseen = 'id';
+ } elsif (s/^\s{10}//) {
+ # 10 leading spaces or more
+ # indicate commit metadata
+
+ # date
+ if ($lastseen eq 'id' && m/^(\d{4}-\d\d-\d\d \d\d:\d\d:\d\d)/){
+ $ps{date} = $1;
+ $lastseen = 'date';
+ } elsif ($_ eq 'merges in:') {
+ $ps{merges} = [];
+ $lastseen = 'merges';
+ } elsif ($lastseen eq 'merges' && s/^\s{2}//) {
+ my $id = $_;
+ push (@{$ps{merges}}, $id);
+
+ # aggressive branch finding:
+ if ($opt_D) {
+ my $branch = extract_versionname($id);
+ my $repo = extract_reponame($branch);
+
+ if (archive_reachable($repo) &&
+ !defined $arch_branches{$branch}) {
+ $arch_branches{$branch} = $stage + 1;
+ }
+ }
+ } else {
+ warn "more metadata after merges!?: $_\n" unless /^\s*$/;
}
-
- $authoremail =~ m/^(.+)\s(\S+)$/;
- $ps{author} = $1;
- $ps{email} = $2;
-
- $lastseen = 'date';
-
- } elsif ($lastseen eq 'date') {
- # the only hint is position
- # subject is after date
- $ps{subj} = $_;
- $lastseen = 'subj';
-
- } elsif ($lastseen eq 'subj' && $_ eq 'merges in:') {
- $ps{merges} = [];
- $lastseen = 'merges';
-
- } elsif ($lastseen eq 'merges' && s/^\s{2}//) {
- push (@{$ps{merges}}, $_);
- } else {
- warn 'more metadata after merges!?';
}
-
}
- }
- if (%ps) {
- my %temp = %ps; # break references
- push (@psets, \%temp);
- $psets{ $temp{id} } = \%temp;
- %ps = ();
- }
- close ABROWSE;
+ if (%ps && !exists $psets{ $ps{id} }) {
+ my %temp = %ps; # break references
+ if (@psets && $psets[$#psets]{branch} eq $ps{branch}) {
+ $temp{parent_id} = $psets[$#psets]{id};
+ }
+ push (@psets, \%temp);
+ $psets{ $temp{id} } = \%temp;
+ }
+
+ close ABROWSE or die "$TLA abrowse failed on $limit\n";
+ }
} # end foreach $root
+do_abrowse(1);
+my $depth = 2;
+$opt_D ||= 0;
+while ($depth <= $opt_D) {
+ do_abrowse($depth);
+ $depth++;
+}
+
## Order patches by time
+# FIXME see if we can find a more optimal way to do this by graphing
+# the ancestry data and walking it, that way we won't have to rely on
+# client-supplied dates
@psets = sort {$a->{date}.$b->{id} cmp $b->{date}.$b->{id}} @psets;
#print Dumper \@psets;
@@ -209,7 +226,7 @@ unless (-d $git_dir) { # initial import
}
} else { # progressing an import
# load the rptags
- opendir(DIR, "$git_dir/archimport/tags")
+ opendir(DIR, $ptag_dir)
|| die "can't opendir: $!";
while (my $file = readdir(DIR)) {
# skip non-interesting-files
@@ -829,6 +846,7 @@ sub find_parents {
}
}
}
+
@parents = ();
foreach (keys %parents) {
push @parents, '-p', ptag($_);
@@ -856,6 +874,7 @@ sub commitid2pset {
return $ps;
}
+
# an alterative to `command` that allows input to be passed as an array
# to work around shell problems with weird characters in arguments
sub safe_pipe_capture {
@@ -869,4 +888,30 @@ sub safe_pipe_capture {
return wantarray ? @output : join('',@output);
}
+# `tla logs -rf -d <dir> | head -n1` or `baz tree-id <dir>`
+sub arch_tree_id {
+ my $dir = shift;
+ chomp( my $ret = (safe_pipe_capture($TLA,'logs','-rf','-d',$dir))[0] );
+ return $ret;
+}
+
+sub archive_reachable {
+ my $archive = shift;
+ return 1 if $reachable{$archive};
+ return 0 if $unreachable{$archive};
+
+ if (system "$TLA whereis-archive $archive >/dev/null") {
+ if ($opt_a && (system($TLA,'register-archive',
+ "http://mirrors.sourcecontrol.net/$archive") == 0)) {
+ $reachable{$archive} = 1;
+ return 1;
+ }
+ print STDERR "Archive is unreachable: $archive\n";
+ $unreachable{$archive} = 1;
+ return 0;
+ } else {
+ $reachable{$archive} = 1;
+ return 1;
+ }
+}
---
0.99.9.GIT
^ permalink raw reply related
* [PATCH 6/9] safer log file parsing
From: Eric Wong @ 2005-11-24 7:53 UTC (permalink / raw)
To: Martin Langhoff; +Cc: git list, Martin Langhoff
In-Reply-To: <20051124075243.GF4789@mail.yhbt.net>
Better logfile parsing, no longer confused by 'headers' after the first
blank line.
Re-enabled tag-reading with abrowse (baz and tla compatible)
Remove need to quote args to external processes
Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
git-archimport.perl | 211 +++++++++++++++++++++++++++------------------------
1 files changed, 112 insertions(+), 99 deletions(-)
applies-to: 1633bcf09400e93aca2eb335181db298a5f49350
3e12af1d958e2d631e27a2f696ca71f83094c7c3
diff --git a/git-archimport.perl b/git-archimport.perl
index 3968740..8676f35 100755
--- a/git-archimport.perl
+++ b/git-archimport.perl
@@ -140,10 +140,10 @@ sub do_abrowse {
$ps{type} = 's';
} elsif ($type =~ /\(.*import\)/) {
$ps{type} = 'i';
- } elsif ($type =~ m/\(tag.*\)/) {
+ } elsif ($type =~ m/\(tag.*?(\S+\@\S+).*?\)/) {
$ps{type} = 't';
# read which revision we've tagged when we parse the log
- #$ps{tag} = $1;
+ $ps{tag} = $1;
} else {
warn "Unknown type $type";
}
@@ -359,78 +359,73 @@ foreach my $ps (@psets) {
#
my $tree;
- my $commitlog = safe_pipe_capture($TLA,'cat-archive-log',$ps->{id});
+ my @commitlog = safe_pipe_capture($TLA,'cat-archive-log',$ps->{id});
die "Error in cat-archive-log: $!" if $?;
- # parselog will git-add/rm files
- # and generally prepare things for the commit
- # NOTE: parselog will shell-quote filenames!
- my ($sum, $msg, $add, $del, $mod, $ren) = parselog($commitlog);
- my $logmessage = "$sum\n$msg";
-
+ parselog($ps,\@commitlog);
# imports don't give us good info
# on added files. Shame on them
- if ($ps->{type} eq 'i' || $ps->{type} eq 't') {
- `find . -type f -print0 | grep -zv '^./$git_dir' | xargs -0 -l100 git-update-index --add`;
- `git-ls-files --deleted -z | xargs --no-run-if-empty -0 -l100 git-update-index --remove`;
+ if ($ps->{type} eq 'i' || $ps->{type} eq 't') {
+ system('git-ls-files --others -z | '.
+ 'git-update-index --add -z --stdin') == 0 or die "$! $?\n";
+ system('git-ls-files --deleted -z | '.
+ 'git-update-index --remove -z --stdin') == 0 or die "$! $?\n";
}
- if (@$add) {
+ # TODO: handle removed_directories and renamed_directories:
+
+ if (my $add = $ps->{new_files}) {
while (@$add) {
my @slice = splice(@$add, 0, 100);
- my $slice = join(' ', @slice);
- `git-update-index --add $slice`;
- die "Error in git-update-index --add: $!" if $?;
+ system('git-update-index','--add','--',@slice) == 0 or
+ die "Error in git-update-index --add: $! $?\n";
}
}
- if (@$del) {
- foreach my $file (@$del) {
- unlink $file or die "Problems deleting $file : $!";
- }
+
+ if (my $del = $ps->{removed_files}) {
+ unlink @$del;
while (@$del) {
my @slice = splice(@$del, 0, 100);
- my $slice = join(' ', @slice);
- `git-update-index --remove $slice`;
- die "Error in git-update-index --remove: $!" if $?;
+ system('git-update-index','--remove','--',@slice) == 0 or
+ die "Error in git-update-index --remove: $! $?\n";
}
}
- if (@$ren) { # renamed
+
+ if (my $ren = $ps->{renamed_files}) { # renamed
if (@$ren % 2) {
die "Odd number of entries in rename!?";
}
- ;
+
while (@$ren) {
- my $from = pop @$ren;
- my $to = pop @$ren;
+ my $from = shift @$ren;
+ my $to = shift @$ren;
unless (-d dirname($to)) {
mkpath(dirname($to)); # will die on err
}
- #print "moving $from $to";
- `mv $from $to`;
- die "Error renaming $from $to : $!" if $?;
- `git-update-index --remove $from`;
- die "Error in git-update-index --remove: $!" if $?;
- `git-update-index --add $to`;
- die "Error in git-update-index --add: $!" if $?;
+ print "moving $from $to";
+ rename($from, $to) or die "Error renaming '$from' '$to': $!\n";
+ system('git-update-index','--remove','--',$from) == 0 or
+ die "Error in git-update-index --remove: $! $?\n";
+ system('git-update-index','--add','--',$to) == 0 or
+ die "Error in git-update-index --add: $! $?\n";
}
}
- if (@$mod) { # must be _after_ renames
+
+ if (my $mod = $ps->{modified_files}) {
while (@$mod) {
my @slice = splice(@$mod, 0, 100);
- my $slice = join(' ', @slice);
- `git-update-index $slice`;
- die "Error in git-update-index: $!" if $?;
+ system('git-update-index','--',@slice) == 0 or
+ die "Error in git-update-index: $! $?\n";
}
}
-
+
# warn "errors when running git-update-index! $!";
$tree = `git-write-tree`;
die "cannot write tree $!" if $?;
chomp $tree;
-
#
# Who's your daddy?
@@ -464,13 +459,14 @@ foreach my $ps (@psets) {
$ENV{GIT_COMMITTER_EMAIL} = $ps->{email};
$ENV{GIT_COMMITTER_DATE} = $ps->{date};
- my ($pid, $commit_rh, $commit_wh);
- $commit_rh = 'commit_rh';
- $commit_wh = 'commit_wh';
-
- $pid = open2(*READER, *WRITER,'git-commit-tree',$tree,@par)
+ my $pid = open2(*READER, *WRITER,'git-commit-tree',$tree,@par)
or die $!;
- print WRITER $logmessage; # write
+ print WRITER $ps->{summary},"\n";
+ print WRITER $ps->{message},"\n";
+
+ # make it easy to backtrack and figure out which Arch revision this was:
+ print WRITER 'git-archimport-id: ',$ps->{id},"\n";
+
close WRITER;
my $commitid = <READER>; # read
chomp $commitid;
@@ -568,7 +564,9 @@ sub apply_cset {
# =for reference
-# A log entry looks like
+# notes: *-files/-directories keys cannot have spaces, they're always
+# pika-escaped. Everything after the first newline
+# A log entry looks like:
# Revision: moodle-org--moodle--1.3.3--patch-15
# Archive: arch-eduforge@catalyst.net.nz--2004
# Creator: Penny Leach <penny@catalyst.net.nz>
@@ -586,70 +584,85 @@ sub apply_cset {
# admin/editor.html backup/lib.php backup/restore.php
# New-patches: arch-eduforge@catalyst.net.nz--2004/moodle-org--moodle--1.3.3--patch-15
# Summary: Updating to latest from MOODLE_14_STABLE (1.4.5+)
+# summary can be multiline with a leading space just like the above fields
# Keywords:
#
# Updating yadda tadda tadda madda
sub parselog {
- my $log = shift;
- #print $log;
-
- my (@add, @del, @mod, @ren, @kw, $sum, $msg );
-
- if ($log =~ m/(?:\n|^)New-files:(.*?)(?=\n\w)/s ) {
- my $files = $1;
- @add = split(m/\s+/s, $files);
- }
-
- if ($log =~ m/(?:\n|^)Removed-files:(.*?)(?=\n\w)/s ) {
- my $files = $1;
- @del = split(m/\s+/s, $files);
- }
-
- if ($log =~ m/(?:\n|^)Modified-files:(.*?)(?=\n\w)/s ) {
- my $files = $1;
- @mod = split(m/\s+/s, $files);
- }
-
- if ($log =~ m/(?:\n|^)Renamed-files:(.*?)(?=\n\w)/s ) {
- my $files = $1;
- @ren = split(m/\s+/s, $files);
- }
-
- $sum ='';
- if ($log =~ m/^Summary:(.+?)$/m ) {
- $sum = $1;
- $sum =~ s/^\s+//;
- $sum =~ s/\s+$//;
- }
+ my ($ps, $log) = @_;
+ my $key = undef;
- $msg = '';
- if ($log =~ m/\n\n(.+)$/s) {
- $msg = $1;
- $msg =~ s/^\s+//;
- $msg =~ s/\s+$//;
+ # headers we want that contain filenames:
+ my %want_headers = (
+ new_files => 1,
+ modified_files => 1,
+ renamed_files => 1,
+ renamed_directories => 1,
+ removed_files => 1,
+ removed_directories => 1,
+ );
+
+ chomp (@$log);
+ while ($_ = shift @$log) {
+ if (/^Continuation-of:\s*(.*)/) {
+ $ps->{tag} = $1;
+ $key = undef;
+ } elsif (/^Summary:\s*(.*)$/ ) {
+ # summary can be multiline as long as it has a leading space
+ $ps->{summary} = [ $1 ];
+ $key = 'summary';
+ } elsif (/^Creator: (.*)\s*<([^\>]+)>/) {
+ $ps->{author} = $1;
+ $ps->{email} = $2;
+ $key = undef;
+ # any *-files or *-directories can be read here:
+ } elsif (/^([A-Z][a-z\-]+):\s*(.*)$/) {
+ my $val = $2;
+ $key = lc $1;
+ $key =~ tr/-/_/; # too lazy to quote :P
+ if ($want_headers{$key}) {
+ push @{$ps->{$key}}, split(/\s+/, $val);
+ } else {
+ $key = undef;
+ }
+ } elsif (/^$/) {
+ last; # remainder of @$log that didn't get shifted off is message
+ } elsif ($key) {
+ if (/^\s+(.*)$/) {
+ if ($key eq 'summary') {
+ push @{$ps->{$key}}, $1;
+ } else { # files/directories:
+ push @{$ps->{$key}}, split(/\s+/, $1);
+ }
+ } else {
+ $key = undef;
+ }
+ }
}
-
-
- # cleanup the arrays
- foreach my $ref ( (\@add, \@del, \@mod, \@ren) ) {
- my @tmp = ();
- while (my $t = pop @$ref) {
- next unless length ($t);
- next if $t =~ m!\{arch\}/!;
- next if $t =~ m!\.arch-ids/!;
- next if $t =~ m!\.arch-inventory$!;
+
+ # post-processing:
+ $ps->{summary} = join("\n",@{$ps->{summary}})."\n";
+ $ps->{message} = join("\n",@$log);
+
+ # skip Arch control files, unescape pika-escaped files
+ foreach my $k (keys %want_headers) {
+ next unless (defined $ps->{$k});
+ my @tmp;
+ foreach my $t (@{$ps->{$k}}) {
+ next unless length ($t);
+ next if $t =~ m!\{arch\}/!;
+ next if $t =~ m!\.arch-ids/!;
+ # should we skip this?
+ next if $t =~ m!\.arch-inventory$!;
# tla cat-archive-log will give us filenames with spaces as file\(sp)name - why?
# we can assume that any filename with \ indicates some pika escaping that we want to get rid of.
- if ($t =~ /\\/ ){
+ if ($t =~ /\\/ ){
$t = (safe_pipe_capture($TLA,'escape','--unescaped',$t))[0];
}
- push (@tmp, $t);
+ push @tmp, $t;
}
- @$ref = @tmp;
+ $ps->{$k} = \@tmp if scalar @tmp;
}
-
- #print Dumper [$sum, $msg, \@add, \@del, \@mod, \@ren];
- return ($sum, $msg, \@add, \@del, \@mod, \@ren);
}
# write/read a tag
---
0.99.9.GIT
^ permalink raw reply related
* [PATCH 7/9] Add the accurate changeset applyer
From: Eric Wong @ 2005-11-24 7:55 UTC (permalink / raw)
To: Martin Langhoff; +Cc: git list, Martin Langhoff
In-Reply-To: <20051124075355.GG4789@mail.yhbt.net>
And make it the default.
This includes stats tracking to verbose mode
Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
git-archimport.perl | 201 ++++++++++++++++++++++++++++++++++++++++++++-------
1 files changed, 172 insertions(+), 29 deletions(-)
applies-to: aa9140057c95e59f65de0794f9054796fbfc96e5
32e5887eedb01ac4c398a06b0a1433ff6f4599fe
diff --git a/git-archimport.perl b/git-archimport.perl
index 8676f35..1cf1261 100755
--- a/git-archimport.perl
+++ b/git-archimport.perl
@@ -25,6 +25,9 @@ See man (1) git-archimport for more deta
- audit shell-escaping of filenames
- hide our private tags somewhere smarter
- find a way to make "cat *patches | patch" safe even when patchfiles are missing newlines
+ - sort and apply patches by graphing ancestry relations instead of just
+ relying in dates supplied in the changeset itself.
+ tla ancestry-graph -m could be helpful here...
=head1 Devel tricks
@@ -66,18 +69,18 @@ my $git_dir = $ENV{"GIT_DIR"} || ".git";
$ENV{"GIT_DIR"} = $git_dir;
my $ptag_dir = "$git_dir/archimport/tags";
-our($opt_h,$opt_v,$opt_T,$opt_t,$opt_D,$opt_a,$opt_o);
+our($opt_h,$opt_f,$opt_v,$opt_T,$opt_t,$opt_D,$opt_a,$opt_o);
sub usage() {
print STDERR <<END;
Usage: ${\basename $0} # fetch/update GIT from Arch
- [ -o ] [ -h ] [ -v ] [ -T ] [ -a ] [ -D depth ] [ -t tempdir ]
+ [ -f ] [ -o ] [ -h ] [ -v ] [ -T ] [ -a ] [ -D depth ] [ -t tempdir ]
repository/arch-branch [ repository/arch-branch] ...
END
exit(1);
}
-getopts("Thvat:D:") or usage();
+getopts("fThvat:D:") or usage();
usage if $opt_h;
@ARGV >= 1 or usage();
@@ -95,6 +98,10 @@ my %reachable = (); # Arch r
my %unreachable = (); # Arch repositories we can't access :<
my @psets = (); # the collection
my %psets = (); # the collection, by name
+my %stats = ( # Track which strategy we used to import:
+ get_tag => 0, replay => 0, get_new => 0, get_delta => 0,
+ simple_changeset => 0, import_or_tag => 0
+);
my %rptags = (); # my reverse private tags
# to map a SHA1 to a commitid
@@ -288,29 +295,69 @@ sub old_style_branchname {
*git_branchname = $opt_o ? *old_style_branchname : *tree_dirname;
-# process patchsets
-foreach my $ps (@psets) {
- $ps->{branch} = git_branchname($ps->{id});
-
- #
- # ensure we have a clean state
- #
- if (`git-diff-files`) {
- die "Unclean tree when about to process $ps->{id} " .
- " - did we fail to commit cleanly before?";
- }
- die $! if $?;
-
- #
- # skip commits already in repo
- #
- if (ptag($ps->{id})) {
- $opt_v && print " * Skipping already imported: $ps->{id}\n";
- next;
+sub process_patchset_accurate {
+ my $ps = shift;
+
+ # switch to that branch if we're not already in that branch:
+ if (-e "$git_dir/refs/heads/$ps->{branch}") {
+ system('git-checkout','-f',$ps->{branch}) == 0 or die "$! $?\n";
+
+ # remove any old stuff that got leftover:
+ my $rm = safe_pipe_capture('git-ls-files','--others','-z');
+ rmtree(split(/\0/,$rm)) if $rm;
}
+
+ # Apply the import/changeset/merge into the working tree
+ my $dir = sync_to_ps($ps);
+ # read the new log entry:
+ my @commitlog = safe_pipe_capture($TLA,'cat-log','-d',$dir,$ps->{id});
+ die "Error in cat-log: $!" if $?;
+ chomp @commitlog;
+
+ # grab variables we want from the log, new fields get added to $ps:
+ # (author, date, email, summary, message body ...)
+ parselog($ps, \@commitlog);
+
+ if ($ps->{id} =~ /--base-0$/ && $ps->{id} ne $psets[0]{id}) {
+ # this should work when importing continuations
+ if ($ps->{tag} && (my $branchpoint = eval { ptag($ps->{tag}) })) {
+
+ # find where we are supposed to branch from
+ system('git-checkout','-f','-b',$ps->{branch},
+ $branchpoint) == 0 or die "$! $?\n";
+
+ # remove any old stuff that got leftover:
+ my $rm = safe_pipe_capture('git-ls-files','--others','-z');
+ rmtree(split(/\0/,$rm)) if $rm;
- print " * Starting to work on $ps->{id}\n";
+ # If we trust Arch with the fact that this is just
+ # a tag, and it does not affect the state of the tree
+ # then we just tag and move on
+ tag($ps->{id}, $branchpoint);
+ ptag($ps->{id}, $branchpoint);
+ print " * Tagged $ps->{id} at $branchpoint\n";
+ return 0;
+ } else {
+ warn "Tagging from unknown id unsupported\n" if $ps->{tag};
+ }
+ # allow multiple bases/imports here since Arch supports cherry-picks
+ # from unrelated trees
+ }
+
+ # update the index with all the changes we got
+ system('git-ls-files --others -z | '.
+ 'git-update-index --add -z --stdin') == 0 or die "$! $?\n";
+ system('git-ls-files --deleted -z | '.
+ 'git-update-index --remove -z --stdin') == 0 or die "$! $?\n";
+ system('git-ls-files -z | '.
+ 'git-update-index -z --stdin') == 0 or die "$! $?\n";
+ return 1;
+}
+# the native changeset processing strategy. This is very fast, but
+# does not handle permissions or any renames involving directories
+sub process_patchset_fast {
+ my $ps = shift;
#
# create the branch if needed
#
@@ -338,7 +385,7 @@ foreach my $ps (@psets) {
tag($ps->{id}, $branchpoint);
ptag($ps->{id}, $branchpoint);
print " * Tagged $ps->{id} at $branchpoint\n";
- next;
+ return 0;
}
die $! if $?;
}
@@ -348,16 +395,17 @@ foreach my $ps (@psets) {
#
if ($ps->{type} eq 'i' || $ps->{type} eq 't') {
apply_import($ps) or die $!;
+ $stats{import_or_tag}++;
$import=0;
} elsif ($ps->{type} eq 's') {
apply_cset($ps);
+ $stats{simple_changeset}++;
}
#
# prepare update git's index, based on what arch knows
# about the pset, resolve parents, etc
#
- my $tree;
my @commitlog = safe_pipe_capture($TLA,'cat-archive-log',$ps->{id});
die "Error in cat-archive-log: $!" if $?;
@@ -404,14 +452,13 @@ foreach my $ps (@psets) {
unless (-d dirname($to)) {
mkpath(dirname($to)); # will die on err
}
- print "moving $from $to";
+ # print "moving $from $to";
rename($from, $to) or die "Error renaming '$from' '$to': $!\n";
system('git-update-index','--remove','--',$from) == 0 or
die "Error in git-update-index --remove: $! $?\n";
system('git-update-index','--add','--',$to) == 0 or
die "Error in git-update-index --add: $! $?\n";
}
-
}
if (my $mod = $ps->{modified_files}) {
@@ -421,9 +468,46 @@ foreach my $ps (@psets) {
die "Error in git-update-index: $! $?\n";
}
}
+ return 1; # we successfully applied the changeset
+}
+
+if ($opt_f) {
+ print "Will import patchsets using the fast strategy\n",
+ "Renamed directories and permission changes will be missed\n";
+ *process_patchset = *process_patchset_fast;
+} else {
+ print "Using the default (accurate) import strategy.\n",
+ "Things may be a bit slow\n";
+ *process_patchset = *process_patchset_accurate;
+}
+
+foreach my $ps (@psets) {
+ # process patchsets
+ $ps->{branch} = git_branchname($ps->{id});
+
+ #
+ # ensure we have a clean state
+ #
+ if (my $dirty = `git-diff-files`) {
+ die "Unclean tree when about to process $ps->{id} " .
+ " - did we fail to commit cleanly before?\n$dirty";
+ }
+ die $! if $?;
+ #
+ # skip commits already in repo
+ #
+ if (ptag($ps->{id})) {
+ $opt_v && print " * Skipping already imported: $ps->{id}\n";
+ return 0;
+ }
+
+ print " * Starting to work on $ps->{id}\n";
+
+ process_patchset($ps) or next;
+
# warn "errors when running git-update-index! $!";
- $tree = `git-write-tree`;
+ my $tree = `git-write-tree`;
die "cannot write tree $!" if $?;
chomp $tree;
@@ -494,6 +578,65 @@ foreach my $ps (@psets) {
print " + commit $commitid\n";
$opt_v && print " + commit date is $ps->{date} \n";
$opt_v && print " + parents: ",join(' ',@par),"\n";
+ if (my $dirty = `git-diff-files`) {
+ die "22 Unclean tree when about to process $ps->{id} " .
+ " - did we fail to commit cleanly before?\n$dirty";
+ }
+}
+
+if ($opt_v) {
+ foreach (sort keys %stats) {
+ print" $_: $stats{$_}\n";
+ }
+}
+exit 0;
+
+# used by the accurate strategy:
+sub sync_to_ps {
+ my $ps = shift;
+ my $tree_dir = $tmp.'/'.tree_dirname($ps->{id});
+
+ $opt_v && print "sync_to_ps($ps->{id}) method: ";
+
+ if (-d $tree_dir) {
+ if ($ps->{type} eq 't') {
+ $opt_v && print "get (tag)\n";
+ # looks like a tag-only or (worse,) a mixed tags/changeset branch,
+ # can't rely on replay to work correctly on these
+ rmtree($tree_dir);
+ safe_pipe_capture($TLA,'get','--no-pristine',$ps->{id},$tree_dir);
+ $stats{get_tag}++;
+ } else {
+ my $tree_id = arch_tree_id($tree_dir);
+ if ($ps->{parent_id} && ($ps->{parent_id} eq $tree_id)) {
+ # the common case (hopefully)
+ $opt_v && print "replay\n";
+ safe_pipe_capture($TLA,'replay','-d',$tree_dir,$ps->{id});
+ $stats{replay}++;
+ } else {
+ # getting one tree is usually faster than getting two trees
+ # and applying the delta ...
+ rmtree($tree_dir);
+ $opt_v && print "apply-delta\n";
+ safe_pipe_capture($TLA,'get','--no-pristine',
+ $ps->{id},$tree_dir);
+ $stats{get_delta}++;
+ }
+ }
+ } else {
+ # new branch work
+ $opt_v && print "get (new tree)\n";
+ safe_pipe_capture($TLA,'get','--no-pristine',$ps->{id},$tree_dir);
+ $stats{get_new}++;
+ }
+
+ # added -I flag to rsync since we're going to fast! AIEEEEE!!!!
+ system('rsync','-aI','--delete','--exclude',$git_dir,
+# '--exclude','.arch-inventory',
+ '--exclude','.arch-ids','--exclude','{arch}',
+ '--exclude','+*','--exclude',',*',
+ "$tree_dir/",'./') == 0 or die "Cannot rsync $tree_dir: $! $?";
+ return $tree_dir;
}
sub apply_import {
@@ -896,7 +1039,7 @@ sub safe_pipe_capture {
@output = (<$child>);
close $child or die join(' ',@_).": $! $?";
} else {
- exec(@_) or die $?; # exec() can fail the executable can't be found
+ exec(@_) or die "$! $?"; # exec() can fail the executable can't be found
}
return wantarray ? @output : join('',@output);
}
---
0.99.9.GIT
^ permalink raw reply related
* [PATCH 8/9] Fix a bug I introduced in the new log parser
From: Eric Wong @ 2005-11-24 7:56 UTC (permalink / raw)
To: Martin Langhoff; +Cc: git list, Martin Langhoff
In-Reply-To: <20051124075504.GH4789@mail.yhbt.net>
This fixes the case (that worked originally in Martin's version)
where the only new/modified files are Arch control files.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
git-archimport.perl | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
applies-to: db1362fe6567e349ff9dd9d70ce23c88a42a0ff2
ebe0689722f6c1440e680ec9a235b3dd571c7de0
diff --git a/git-archimport.perl b/git-archimport.perl
index 1cf1261..0080850 100755
--- a/git-archimport.perl
+++ b/git-archimport.perl
@@ -790,7 +790,7 @@ sub parselog {
# skip Arch control files, unescape pika-escaped files
foreach my $k (keys %want_headers) {
next unless (defined $ps->{$k});
- my @tmp;
+ my @tmp = ();
foreach my $t (@{$ps->{$k}}) {
next unless length ($t);
next if $t =~ m!\{arch\}/!;
@@ -804,7 +804,7 @@ sub parselog {
}
push @tmp, $t;
}
- $ps->{$k} = \@tmp if scalar @tmp;
+ $ps->{$k} = \@tmp;
}
}
---
0.99.9.GIT
^ permalink raw reply related
* [PATCH 9/9] fix a in new changeset applyer addition
From: Eric Wong @ 2005-11-24 7:58 UTC (permalink / raw)
To: Martin Langhoff; +Cc: git list, Martin Langhoff
In-Reply-To: <20051124075631.GI4789@mail.yhbt.net>
Fix a stupid bug I introduced when splitting the
accurate and fast changeset appliers.
Also, remove an old debugging statement I added
Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
git-archimport.perl | 6 +-----
1 files changed, 1 insertions(+), 5 deletions(-)
applies-to: 6dfed0cb7c209cf47902d6dfcd02a974d252041b
b081cb1e0f79f1a290bcf1f2161d63415ec5e2a9
diff --git a/git-archimport.perl b/git-archimport.perl
index 0080850..aab4e38 100755
--- a/git-archimport.perl
+++ b/git-archimport.perl
@@ -499,7 +499,7 @@ foreach my $ps (@psets) {
#
if (ptag($ps->{id})) {
$opt_v && print " * Skipping already imported: $ps->{id}\n";
- return 0;
+ next;
}
print " * Starting to work on $ps->{id}\n";
@@ -578,10 +578,6 @@ foreach my $ps (@psets) {
print " + commit $commitid\n";
$opt_v && print " + commit date is $ps->{date} \n";
$opt_v && print " + parents: ",join(' ',@par),"\n";
- if (my $dirty = `git-diff-files`) {
- die "22 Unclean tree when about to process $ps->{id} " .
- " - did we fail to commit cleanly before?\n$dirty";
- }
}
if ($opt_v) {
---
0.99.9.GIT
^ permalink raw reply related
* Re: Get rid of .git/branches/ and .git/remotes/?
From: Andreas Ericsson @ 2005-11-24 8:05 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.63.0511240042350.11106@wbgn013.biozentrum.uni-wuerzburg.de>
Johannes Schindelin wrote:
> Hi,
>
> On Thu, 24 Nov 2005, Andreas Ericsson wrote:
>
>
>>git-repo-config is way friendlier for us poor lazy folks. That last 'y' in
>>"repository" really breaks the flow.
>
>
> How about "git-local-config"?
>
Nah. It would really mess things up if there's ever such a thing as a
real local config (ie, ~/.gitrc).
git-config-set handles configuration for a particular repo, so
git-repo-config is better, IMO.
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
^ permalink raw reply
* Re: Why not clone to a remote directory over SSH
From: Andreas Ericsson @ 2005-11-24 8:14 UTC (permalink / raw)
To: Nick Hengeveld; +Cc: Johannes Schindelin, Junio C Hamano, Carl Baldwin, git
In-Reply-To: <20051124001955.GP3968@reactrix.com>
Nick Hengeveld wrote:
> On Thu, Nov 24, 2005 at 12:41:56AM +0100, Johannes Schindelin wrote:
>
>
>>Obviously, this only works if the other side can connect to this side via
>>ssh. Hmm. Firewalls? Disabled sshd? `hostname` not reliably returning a
>>valid address for the remote side?
>
>
> If sshd is running locally, you can forward a port back to yourself and
> have the remote clone localhost:`pwd`. Assuming there's a way to clone
> using a nonstandard port.
>
ssh://host.xz:port/path/to/repo.git
should work. If it doesn't I'll fix it.
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
^ permalink raw reply
* Re: [PATCH 4/9] remove git wrapper dependency
From: Andreas Ericsson @ 2005-11-24 8:20 UTC (permalink / raw)
To: git list
In-Reply-To: <20051124075133.GE4789@mail.yhbt.net>
Eric Wong wrote:
> use git-diff-files instead of git diff-files so we don't rely on the
> wrapper being installed (some people may have git as GNU interactive
> tools :)
>
This one should do
git --exec-path
first to get the proper path to git-diff-files. Fall back to it being in
the path if finding out fails.
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
^ permalink raw reply
* Re: Get rid of .git/branches/ and .git/remotes/?
From: Junio C Hamano @ 2005-11-24 8:33 UTC (permalink / raw)
To: Andreas Ericsson, Johannes Schindelin; +Cc: git
In-Reply-To: <43857430.7060103@op5.se>
Andreas Ericsson <ae@op5.se> writes:
> git-config-set handles configuration for a particular repo, so
> git-repo-config is better, IMO.
Ok, so git-repo-config that is. Johannes?
^ permalink raw reply
* Re: [PATCH 4/9] remove git wrapper dependency
From: Junio C Hamano @ 2005-11-24 8:35 UTC (permalink / raw)
To: git
In-Reply-To: <438577E0.30109@op5.se>
Andreas Ericsson <ae@op5.se> writes:
> Eric Wong wrote:
>> use git-diff-files instead of git diff-files so we don't rely on the
>> wrapper being installed (some people may have git as GNU interactive
>> tools :)
>>
>
> This one should do
> git --exec-path
>
> first to get the proper path to git-diff-files. Fall back to it being in
> the path if finding out fails.
Eric is worried about the case where git on your PATH is GNU
interactive tools, so "git --exec-path" would not give you what
you want ;-).
^ permalink raw reply
* Re: files are disappearing in git
From: Nico -telmich- Schottelius @ 2005-11-24 8:46 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Git ML
In-Reply-To: <Pine.LNX.4.64.0511230917130.13959@g5.osdl.org>
[-- Attachment #1: Type: text/plain, Size: 2720 bytes --]
Linus Torvalds [Wed, Nov 23, 2005 at 09:20:28AM -0800]:
> On Wed, 23 Nov 2005, Nico -telmich- Schottelius wrote:
> >
> > I've the problem that some files (a directory with 3 files) is simply 'away':
> >
> > - We added it once
> > - In current tree it's away
> > - pasky aided me in irc to find the commit where it is gone with git bisect
> > --> very nice tool
> > - the commit, after which the directory was gone did NOT modify this directory
> > - though the directory is gone
> >
> > What should I do know to find out what's the reason git 'forgot' that directory?
>
> I bet somebody just messed up the index before that commit.
What would be the best things to corrupt the index?
Our developers here do the following each day:
- cg-update
- <work>
- cg-commit
- cg-update + merge if there are changes
- cg-push origin
--> this goes to NFS mounted /home (in which developers also work)
[9:34] klapperwachstum:walderlift% cat .git/branches/origin
/home/server/git/walderlift.git
> So the commit
> probably _did_ modify the directory, though some incorrect patching or
> some scripting bug.
Hrm, perhaps this is true, though I am not aware _how_ they (two people
besides me) could even manage to do that. Our developers are
Kylix developers under Linux, who are raelly limited to the above
cg-commands and using gitweb for seeing changes.
> If you can guess at all what went on at around that commit (what else
> happened there?) that might help.
In this commit where specified three files changed. The commit text is not related
to the directory which is missing.
Files changed are
Code/Components/Utilities/LW1Calendar.pas
Code/lw1/Client/Terminierung/FeinTerminierungForm.pas
Code/lw1/Client/Terminierung/FeinTerminierungForm.xfm
and files missing after this release are
Code/Spikes/Statistik/
If I select "history" of one of the files at the point it still existed in the tree,
it has only one entry, where it was added.
> Was it a merge?
No, we also assumed that first, but git bisect showed that it is not.
> Is the tree public so that we can look at it and perhaps make a guess from
> what happened around it?
I am sorry it is not. Perhaps I can convince my boss to allow access to it for
some git developers, so someone could debug it. Thought, every information
found in the tree would have to be treated confidental.
Do you have some other hints on how to debug that? Perhaps some conistency checking
tool?
Or perhaps should I put that git directory under cvs? ;-)
Nico
--
Latest project: cinit-0.2.1 (http://linux.schottelius.org/cinit/)
Open Source nutures open minds and free, creative developers.
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 827 bytes --]
^ permalink raw reply
* Re: [PATCH 4/9] remove git wrapper dependency
From: Eric Wong @ 2005-11-24 8:50 UTC (permalink / raw)
To: git
In-Reply-To: <7voe4awii5.fsf@assigned-by-dhcp.cox.net>
Junio C Hamano <junkio@cox.net> wrote:
> Andreas Ericsson <ae@op5.se> writes:
>
> > Eric Wong wrote:
> >> use git-diff-files instead of git diff-files so we don't rely on the
> >> wrapper being installed (some people may have git as GNU interactive
> >> tools :)
> >>
> >
> > This one should do
> > git --exec-path
> >
> > first to get the proper path to git-diff-files. Fall back to it being in
> > the path if finding out fails.
>
> Eric is worried about the case where git on your PATH is GNU
> interactive tools, so "git --exec-path" would not give you what
> you want ;-).
Right on. I'm actually not a GNU interactive tools user, but I do have
empathy for them being a cg (cgvg) user myself for many, many years.
--
Eric Wong
^ permalink raw reply
* Re: files are disappearing in git
From: Junio C Hamano @ 2005-11-24 9:15 UTC (permalink / raw)
To: Nico -telmich- Schottelius; +Cc: git
In-Reply-To: <20051124084633.GA3361@schottelius.org>
Nico -telmich- Schottelius <nico-linux-git@schottelius.org> writes:
> Linus Torvalds [Wed, Nov 23, 2005 at 09:20:28AM -0800]:
>> On Wed, 23 Nov 2005, Nico -telmich- Schottelius wrote:
>>...
>> I bet somebody just messed up the index before that commit.
>
> What would be the best things to corrupt the index?
> Our developers here do the following each day:
>
> - cg-update
> - <work>
> - cg-commit
> - cg-update + merge if there are changes
> - cg-push origin
Was any of the CG:F line changed/removed by the developer during
cg-commit?
Does any of the files under Code/Spikes/Statistik/ have funny
characters in their pathnames?
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox