* Re: Looking up objects that point to other objects
From: Jeff King @ 2011-08-26 20:10 UTC (permalink / raw)
To: Ævar Arnfjörð Bjarmason; +Cc: Git Mailing List
In-Reply-To: <CACBZZX6sydEmuwj_C-KNocjra=6ynud5KFoezPd_Rr3bN4wh2w@mail.gmail.com>
On Fri, Aug 26, 2011 at 09:01:22PM +0200, Ævar Arnfjörð Bjarmason wrote:
> Here's a couple of tasks that require brute-force with the Git object
> format that I've wanted to do at some point.
>
> * Associate a blob with trees
>
> Given a blob sha1 find trees that reference it.
>
> * Associate trees with commits / other trees.
>
> Given a tree find which commit points to that tree, or a parent
> tree N levels up the stack that a commit points to.
I've more frequently wanted to find the entrance and exit points of a
particular blob in history, and used something like:
git log --all --no-abbrev -c --raw --format='commit %H' |
perl -le '
my @blobs = map { qr/$_/ } @ARGV;
while(<STDIN>) {
if (/^commit (.*)/) {
$commit = $1;
}
else {
foreach my $re (@blobs) {
next unless /$re/;
print $commit;
last;
}
}
}
' $blobs
which is fairly efficient. It's brute-force, but at least it all happens
in O(1) processes. It can find blobs in git.git in about a minute or so
on my machine.
I don't think there's a way to ask for all of the trees in a commit in a
single process. It wouldn't be hard to write in C, of course, but it's
not something the current tools support. However, you can use the above
script to narrow the range of commits that you know contain a blob, and
then individually run ls-tree each one. It's better at least than running
ls-tree on every commit in the repo.
Anything that iterates over commits is going to end up seeing the same
trees again and again. I think you could probably do better by thinking
of it like a directed graph problem. Nodes are sha1s, and edges are any
references of interest:
1. For a commit, make an edge from the commit to its tree.
2. For a tree, make an edge from the tree to each of its entries.
And then the problem is reduced to "find all commit nodes that have a
path to $blob". Which you can do by breadth-first search from the
commits (or backwards from the blob).
-Peff
^ permalink raw reply
* Re: [PATCH] gitweb: highlight: strip non-printable characters via col(1)
From: Jakub Narebski @ 2011-08-26 19:54 UTC (permalink / raw)
To: Christopher M. Fuhrman; +Cc: gitster, git, cwilson, sylvain
In-Reply-To: <1314053923-13122-1-git-send-email-cfuhrman@panix.com>
On Tue, 23 Aug 2011, Christopher M. Fuhrman wrote:
> The current code, as is, passes control characters, such as form-feed
> (^L) to highlight which then passes it through to the browser. This
> will cause the browser to display one of the following warnings:
>
> Safari v5.1 (6534.50) & Google Chrome v13.0.782.112:
>
> This page contains the following errors:
>
> error on line 657 at column 38: PCDATA invalid Char value 12
> Below is a rendering of the page up to the first error.
>
> Mozilla Firefox 3.6.19 & Mozilla Firefox 5.0:
>
> XML Parsing Error: not well-formed
> Location:
> http://path/to/git/repo/blah/blah
>
> Both errors were generated by gitweb.perl v1.7.3.4 w/ highlight 2.7
> using arch/ia64/kernel/unwind.c from the Linux kernel.
>
> Strip non-printable control-characters by piping the output produced
> by git-cat-file(1) to col(1) as follows:
>
> git cat-file blob deadbeef314159 | col -bx | highlight <args>
>
> Note usage of the '-x' option which tells col(1) to output multiple
> spaces instead of tabs.
Why use external program (which ming be not installed, or might not
strip control-characters), instead of making gitweb sanitize highlighter
output itself. Something like the patch below (which additionally
shows where there are control characters):
-- >8 --
diff --git i/gitweb/gitweb.perl w/gitweb/gitweb.perl
index 7cf12af..192db2c 100755
--- i/gitweb/gitweb.perl
+++ w/gitweb/gitweb.perl
@@ -1517,6 +1517,17 @@ sub esc_path {
return $str;
}
+# Sanitize for use in XHTML + application/xml+xhtml
+sub sanitize {
+ my $str = shift;
+
+ return undef unless defined $str;
+
+ $str = to_utf8($str);
+ $str =~ s|([[:cntrl:]])|quot_cec($1)|eg;
+ return $str;
+}
+
# Make control characters "printable", using character escape codes (CEC)
sub quot_cec {
my $cntrl = shift;
@@ -6546,7 +6557,8 @@ sub git_blob {
$nr++;
$line = untabify($line);
printf qq!<div class="pre"><a id="l%i" href="%s#l%i" class="linenr">%4i</a> %s</div>\n!,
- $nr, esc_attr(href(-replay => 1)), $nr, $nr, $syntax ? to_utf8($line) : esc_html($line, -nbsp=>1);
+ $nr, esc_attr(href(-replay => 1)), $nr, $nr,
+ $syntax ? sanitize($line) : esc_html($line, -nbsp=>1);
}
}
close $fd
-- 8< --
--
Jakub Narebski
Poland
^ permalink raw reply related
* [PATCH v2/RFC] submodule: Search for merges only at end of recursive merge
From: Brad King @ 2011-08-26 19:30 UTC (permalink / raw)
To: git; +Cc: gitster, Heiko Voigt
In-Reply-To: <7vr548c7un.fsf@alter.siamese.dyndns.org>
The submodule merge search is not useful during virtual merges because
the results cannot be used automatically. Furthermore any suggestions
made by the search may apply to commits different than HEAD:sub and
MERGE_HEAD:sub, thus confusing the user. Skip searching for submodule
merges during a virtual merge such as that between B and C while merging
the heads of:
B---BC
/ \ /
A X
\ / \
C---CB
Run the search only when the recursion level is zero (!o->call_depth).
This fixes known breakage tested in t7405-submodule-merge.
Signed-off-by: Brad King <brad.king@kitware.com>
---
On 8/26/2011 3:04 PM, Junio C Hamano wrote:
> This is a knee-jerk reaction without thinking things thoroughly through,
> but wouldn't it make more sense to do this by conditionally calling
> merge_submodule() when !o->call_depth, leaving the callee oblivious to
> what is in the "merge_options" structure? That way, you do not have to
> touch submodule.c at all, I would think.
I originally considered that but I think merge_submodule is still useful
during virtual merges in the fast forward case. I haven't thought that
through in detail though.
> After all, merge_submodule() should be usable in a future merge strategy
> that is different from recursive and has no notion of call_depth.
That's a worthwhile goal. Perhaps instead we should pass in a parameter
to tell merge_submodule whether or not to do the search after the fast
forward case fails.
Brad
merge-recursive.c | 6 ++++--
submodule.c | 6 +++++-
submodule.h | 2 +-
t/t7405-submodule-merge.sh | 2 +-
4 files changed, 11 insertions(+), 5 deletions(-)
diff --git a/merge-recursive.c b/merge-recursive.c
index 0cc1e6f..390811e 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -830,8 +830,10 @@ static struct merge_file_info merge_file(struct merge_options *o,
free(result_buf.ptr);
result.clean = (merge_status == 0);
} else if (S_ISGITLINK(a->mode)) {
- result.clean = merge_submodule(result.sha, one->path, one->sha1,
- a->sha1, b->sha1);
+ result.clean = merge_submodule(result.sha,
+ one->path, one->sha1,
+ a->sha1, b->sha1,
+ !o->call_depth);
} else if (S_ISLNK(a->mode)) {
hashcpy(result.sha, a->sha1);
diff --git a/submodule.c b/submodule.c
index 1ba9646..bf4f693 100644
--- a/submodule.c
+++ b/submodule.c
@@ -644,7 +644,7 @@ static void print_commit(struct commit *commit)
int merge_submodule(unsigned char result[20], const char *path,
const unsigned char base[20], const unsigned char a[20],
- const unsigned char b[20])
+ const unsigned char b[20], int search)
{
struct commit *commit_base, *commit_a, *commit_b;
int parent_count;
@@ -699,6 +699,10 @@ int merge_submodule(unsigned char result[20], const char *path,
* user needs to confirm the resolution.
*/
+ /* Skip the search if makes no sense to the calling context. */
+ if (!search)
+ return 0;
+
/* find commit which merges them */
parent_count = find_first_merges(&merges, path, commit_a, commit_b);
switch (parent_count) {
diff --git a/submodule.h b/submodule.h
index 5350b0d..d4344c8 100644
--- a/submodule.h
+++ b/submodule.h
@@ -28,6 +28,6 @@ int fetch_populated_submodules(int num_options, const char **options,
int quiet);
unsigned is_submodule_modified(const char *path, int ignore_untracked);
int merge_submodule(unsigned char result[20], const char *path, const unsigned char base[20],
- const unsigned char a[20], const unsigned char b[20]);
+ const unsigned char a[20], const unsigned char b[20], int search);
#endif
diff --git a/t/t7405-submodule-merge.sh b/t/t7405-submodule-merge.sh
index 14da2e3..0d5b42a 100755
--- a/t/t7405-submodule-merge.sh
+++ b/t/t7405-submodule-merge.sh
@@ -269,7 +269,7 @@ test_expect_success 'setup for recursive merge with submodule' '
'
# merge should leave submodule unmerged in index
-test_expect_failure 'recursive merge with submodule' '
+test_expect_success 'recursive merge with submodule' '
(cd merge-recursive &&
test_must_fail git merge top-bc &&
echo "160000 $(git rev-parse top-cb:sub) 2 sub" > expect2 &&
--
1.7.4.4
^ permalink raw reply related
* Re: [PATCH/RFC] submodule: Search for merges only at end of recursive merge
From: Junio C Hamano @ 2011-08-26 19:04 UTC (permalink / raw)
To: Brad King; +Cc: git, Heiko Voigt
In-Reply-To: <03300e44101092641810b34086738a151e01d8ee.1314368109.git.brad.king@kitware.com>
This is a knee-jerk reaction without thinking things thoroughly through,
but wouldn't it make more sense to do this by conditionally calling
merge_submodule() when !o->call_depth, leaving the callee oblivious to
what is in the "merge_options" structure? That way, you do not have to
touch submodule.c at all, I would think.
After all, merge_submodule() should be usable in a future merge strategy
that is different from recursive and has no notion of call_depth.
^ permalink raw reply
* Looking up objects that point to other objects
From: Ævar Arnfjörð Bjarmason @ 2011-08-26 19:01 UTC (permalink / raw)
To: Git Mailing List
Here's a couple of tasks that require brute-force with the Git object
format that I've wanted to do at some point.
* Associate a blob with trees
Given a blob sha1 find trees that reference it.
* Associate trees with commits / other trees.
Given a tree find which commit points to that tree, or a parent
tree N levels up the stack that a commit points to.
Has anyone written tools to do this? They'd obviously be very CPU and
I/O intensive, but occasionally I encounter cases where I'd find this
useful, e.g. to find what commit contains this huge blob, or what
trees / commits are involved with a corrupted object.
^ permalink raw reply
* Re: [PATCH 1/2] fast-import: initialize variable require_explicit_termination
From: Junio C Hamano @ 2011-08-26 18:55 UTC (permalink / raw)
To: Matthieu Moy; +Cc: git
In-Reply-To: <vpqr548hx40.fsf@bauges.imag.fr>
Matthieu Moy <Matthieu.Moy@grenoble-inp.fr> writes:
>> Please do not write unnecessary " = 0" there.
>
> I prefer being explicit, but that's a matter of taste, and I don't
> really care, so let's drop this patch.
I prefer being explicit, too, but "static int foo;" is explicit enough if
you know C.
^ permalink raw reply
* Re: [PATCHv2 5/5] branch: allow pattern arguments
From: Junio C Hamano @ 2011-08-26 18:45 UTC (permalink / raw)
To: Michael J Gruber; +Cc: git, Jeff King
In-Reply-To: <b4a43ac3686d66b3ef8eddbed2c98c56b6f13312.1314367414.git.git@drmicha.warpmail.net>
This and [4/5] both looked good; will update mg/branch-list with this
round.
Thanks.
^ permalink raw reply
* Re: [PATCH 1/2] fast-import: initialize variable require_explicit_termination
From: Matthieu Moy @ 2011-08-26 17:59 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7v8vqgdpsv.fsf@alter.siamese.dyndns.org>
Junio C Hamano <gitster@pobox.com> writes:
> Matthieu Moy <Matthieu.Moy@imag.fr> writes:
>
>> The uninitialized variable seems harmless in practice, but let's still be clean.
>
> It is not "in practice", but by definition, file scope "static int"
> variables are initialized to 0 by the C language
OK, I didn't know this was specified in C, but you're right.
> Please do not write unnecessary " = 0" there.
I prefer being explicit, but that's a matter of taste, and I don't
really care, so let's drop this patch.
--
Matthieu Moy
http://www-verimag.imag.fr/~moy/
^ permalink raw reply
* [PATCH v5] Add a remote helper to interact with mediawiki (fetch & push)
From: Matthieu Moy @ 2011-08-26 17:55 UTC (permalink / raw)
To: git, gitster
Cc: Jeremie Nikaes, Arnaud Lacurie, Claire Fousse, David Amouyal,
Matthieu Moy, Sylvain Boulmé, Matthieu Moy
In-Reply-To: <1314378689-8997-2-git-send-email-Matthieu.Moy@imag.fr>
From: Jeremie Nikaes <jeremie.nikaes@ensimag.imag.fr>
Implement a gate between git and mediawiki, allowing git users to push
and pull objects from mediawiki just as one would do with a classic git
repository thanks to remote-helpers.
The following packages need to be installed (available on common
repositories):
libmediawiki-api-perl
libdatetime-format-iso8601-perl
Use remote helpers in order to be as transparent as possible to the git
user.
Download Mediawiki revisions through the Mediawiki API and then
fast-import into git.
Mediawiki revision number and git commits are linked thanks to notes
bound to commits.
The import part is done on a refs/mediawiki/<remote> branch before
coming to refs/remote/origin/master (Huge thanks to Jonathan Nieder
for his help)
We use UTF-8 everywhere: use encoding 'utf8'; does most of the job, but
we also read the output of Git commands in UTF-8 with the small helper
run_git, and write to the console (STDERR) in UTF-8. This allows a
seamless use of non-ascii characters in page titles, but hasn't been
tested on non-UTF-8 systems. In particular, UTF-8 encoding for filenames
could raise problems if different file systems handle UTF-8 filenames
differently. A uri_escape of mediawiki filenames could be imaginable, and
is still to be discussed further.
Partial cloning is supported using one of:
git clone -c remote.origin.pages='A_Page Another_Page' mediawiki::http://wikiurl
git clone -c remote.origin.categories='Some_Category' mediawiki::http://wikiurl
git clone -c remote.origin.shallow='True' mediawiki::http://wikiurl
Thanks to notes metadata, it is possible to compare remote and local last
mediawiki revision to warn non-fast forward pushes and "everything
up-to-date" case.
When allowed, push looks for each commit between remotes/origin/master
and HEAD, catches every blob related to these commit and push them in
chronological order. To do so, it uses git rev-list --children HEAD and
travels the tree from remotes/origin/master to HEAD through children. In
other words :
* Shortest path from remotes/origin/master to HEAD
* For each commit encountered, push blobs related to this commit
Signed-off-by: Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>
Signed-off-by: Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>
Signed-off-by: Claire Fousse <claire.fousse@ensimag.imag.fr>
Signed-off-by: David Amouyal <david.amouyal@ensimag.imag.fr>
Signed-off-by: Matthieu Moy <matthieu.moy@grenoble-inp.fr>
Signed-off-by: Sylvain Boulmé <sylvain.boulme@imag.fr>
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
Oops, I had an uncommited fix when I sent the previous patch. Change
since v4 is just this:
- if ($fetch_from == 1) {
- if ($n != 0) {
- print STDOUT "reset $ref\n";
- print STDOUT "from :$n\n";
- } else {
- print STDERR "You appear to have cloned an empty mediawiki\n";
- # Something has to be done remote-helper side. If nothing is done, an error is
- # thrown saying that HEAD is refering to unknown object 0000000000000000000
- }
+ if ($fetch_from == 1 && $n == 0) {
+ print STDERR "You appear to have cloned an empty MediaWiki.\n";
+ # Something has to be done remote-helper side. If nothing is done, an error is
+ # thrown saying that HEAD is refering to unknown object 0000000000000000000
+ # and the clone fails.
Sorry for the noise.
contrib/mw-to-git/git-remote-mediawiki | 718 ++++++++++++++++++++++++++++
contrib/mw-to-git/git-remote-mediawiki.txt | 7 +
2 files changed, 725 insertions(+), 0 deletions(-)
create mode 100755 contrib/mw-to-git/git-remote-mediawiki
create mode 100644 contrib/mw-to-git/git-remote-mediawiki.txt
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
new file mode 100755
index 0000000..e24a640
--- /dev/null
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -0,0 +1,718 @@
+#! /usr/bin/perl
+
+# Copyright (C) 2011
+# Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>
+# Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>
+# Claire Fousse <claire.fousse@ensimag.imag.fr>
+# David Amouyal <david.amouyal@ensimag.imag.fr>
+# Matthieu Moy <matthieu.moy@grenoble-inp.fr>
+# License: GPL v2 or later
+
+# Gateway between Git and MediaWiki.
+# https://github.com/Bibzball/Git-Mediawiki/wiki
+#
+# Known limitations:
+#
+# - Only wiki pages are managed, no support for [[File:...]]
+# attachments.
+#
+# - Poor performance in the best case: it takes forever to check
+# whether we're up-to-date (on fetch or push) or to fetch a few
+# revisions from a large wiki, because we use exclusively a
+# page-based synchronization. We could switch to a wiki-wide
+# synchronization when the synchronization involves few revisions
+# but the wiki is large.
+#
+# - Git renames could be turned into MediaWiki renames (see TODO
+# below)
+#
+# - login/password support requires the user to write the password
+# cleartext in a file (see TODO below).
+#
+# - No way to import "one page, and all pages included in it"
+#
+# - Multiple remote MediaWikis have not been very well tested.
+
+use strict;
+use MediaWiki::API;
+use DateTime::Format::ISO8601;
+use encoding 'utf8';
+
+# use encoding 'utf8' doesn't change STDERROR
+# but we're going to output UTF-8 filenames to STDERR
+binmode STDERR, ":utf8";
+
+use URI::Escape;
+use warnings;
+
+# Mediawiki filenames can contain forward slashes. This variable decides by which pattern they should be replaced
+use constant SLASH_REPLACEMENT => "%2F";
+
+# It's not always possible to delete pages (may require some
+# priviledges). Deleted pages are replaced with this content.
+use constant DELETED_CONTENT => "[[Category:Deleted]]\n";
+
+# It's not possible to create empty pages. New empty files in Git are
+# sent with this content instead.
+use constant EMPTY_CONTENT => "<!-- empty page -->\n";
+
+# used to reflect file creation or deletion in diff.
+use constant NULL_SHA1 => "0000000000000000000000000000000000000000";
+
+my $remotename = $ARGV[0];
+my $url = $ARGV[1];
+
+# Accept both space-separated and multiple keys in config file.
+# Spaces should be written as _ anyway because we'll use chomp.
+my @tracked_pages = split(/[ \n]/, run_git("config --get-all remote.". $remotename .".pages"));
+chomp(@tracked_pages);
+
+# Just like @tracked_pages, but for MediaWiki categories.
+my @tracked_categories = split(/[ \n]/, run_git("config --get-all remote.". $remotename .".categories"));
+chomp(@tracked_categories);
+
+my $wiki_login = run_git("config --get remote.". $remotename .".mwLogin");
+# TODO: ideally, this should be able to read from keyboard, but we're
+# inside a remote helper, so our stdin is connect to git, not to a
+# terminal.
+my $wiki_passwd = run_git("config --get remote.". $remotename .".mwPassword");
+chomp ($wiki_login);
+chomp ($wiki_passwd);
+
+# Import only last revisions (both for clone and fetch)
+my $shallow_import = run_git("config --get --bool remote.". $remotename .".shallow");
+chomp($shallow_import);
+$shallow_import = ($shallow_import eq "true");
+
+my $wiki_name = $url;
+$wiki_name =~ s/[^\/]*:\/\///;
+
+my $import_started;
+
+# Commands parser
+my $entry;
+my @cmd;
+while (<STDIN>) {
+ chomp;
+ @cmd = split(/ /);
+ if (defined($cmd[0])) {
+ # Line not blank
+ if ($cmd[0] eq "capabilities") {
+ die("Too many arguments for capabilities") unless (!defined($cmd[1]));
+ mw_capabilities();
+ } elsif ($cmd[0] eq "list") {
+ die("Too many arguments for list") unless (!defined($cmd[2]));
+ mw_list($cmd[1]);
+ } elsif ($cmd[0] eq "import") {
+ die("Invalid arguments for import") unless ($cmd[1] ne "" && !defined($cmd[2]));
+ mw_import($cmd[1]);
+ } elsif ($cmd[0] eq "option") {
+ die("Too many arguments for option") unless ($cmd[1] ne "" && $cmd[2] ne "" && !defined($cmd[3]));
+ mw_option($cmd[1],$cmd[2]);
+ } elsif ($cmd[0] eq "push") {
+ # Check the pattern <src>:<dst>
+ my @pushargs = split(/:/,$cmd[1]);
+ die("Invalid arguments for push") unless ($pushargs[1] ne "" && !defined($pushargs[2]));
+ mw_push($pushargs[0],$pushargs[1]);
+ } else {
+ print STDERR "Unknown command. Aborting...\n";
+ last;
+ }
+ } else {
+ # blank line: we should terminate
+ last;
+ }
+
+ BEGIN { $| = 1 } # flush STDOUT, to make sure the previous
+ # command is fully processed.
+}
+# End of input
+if ($import_started) {
+ # Terminate the fast-import stream properly.
+ # Git requires one "done" command, and only
+ # one This is OK since we only have one
+ # branch, so import will be called only once
+ # (plus once for HEAD, for which we won't
+ # reach this point).
+ print STDOUT "done\n";
+}
+BEGIN { $| = 1 };
+if (!eof(STDIN)) {
+ # Wait for Git to terminate. If we don't, git fetch
+ # (transport-helper.c's sendline function) will try to write
+ # to our stdin, which will be closed, and git fetch will be
+ # killed. That's probably a bug in transport-helper.c, but in
+ # the meantime ...
+ sleep .1;
+};
+
+########################## Functions ##############################
+
+# MediaWiki API instance, created lazily.
+my $mediawiki;
+
+sub mw_connect_maybe {
+ if ($mediawiki) {
+ return;
+ }
+ $mediawiki = MediaWiki::API->new;
+ $mediawiki->{config}->{api_url} = "$url/api.php";
+ if ($wiki_login) {
+ if (!$mediawiki->login({
+ lgname => $wiki_login,
+ lgpassword => $wiki_passwd,
+ })) {
+ print STDERR "Failed to log in mediawiki user \"$wiki_login\" on $url\n";
+ print STDERR "(error " .
+ $mediawiki->{error}->{code} . ': ' .
+ $mediawiki->{error}->{details} . ")\n";
+ exit 1;
+ } else {
+ print STDERR "Logged in with user \"$wiki_login\".\n";
+ }
+ }
+}
+
+sub get_mw_first_pages {
+ my $some_pages = shift;
+ my @some_pages = @{$some_pages};
+
+ my $pages = shift;
+
+ # pattern 'page1|page2|...' required by the API
+ my $titles = join('|', @some_pages);
+
+ my $mw_pages = $mediawiki->api({
+ action => 'query',
+ titles => $titles,
+ });
+ if (!defined($mw_pages)) {
+ print STDERR "fatal: could not query the list of wiki pages.\n";
+ print STDERR "fatal: '$url' does not appear to be a mediawiki\n";
+ print STDERR "fatal: make sure '$url/api.php' is a valid page.\n";
+ exit 1;
+ }
+ while (my ($id, $page) = each (%{$mw_pages->{query}->{pages}})) {
+ if ($id < 0) {
+ print STDERR "Warning: page $page->{title} not found on wiki\n";
+ } else {
+ $pages->{$page->{title}} = $page;
+ }
+ }
+}
+
+sub get_mw_pages {
+ mw_connect_maybe();
+
+ my %pages; # hash on page titles to avoid duplicates
+ my $user_defined;
+ if (@tracked_pages) {
+ $user_defined = 1;
+ # The user provided a list of pages titles, but we
+ # still need to query the API to get the page IDs.
+
+ my @some_pages = @tracked_pages;
+ while (@some_pages) {
+ my $last = 50;
+ if ($#some_pages < $last) {
+ $last = $#some_pages;
+ }
+ my @slice = @some_pages[0..$last];
+ get_mw_first_pages(\@slice, \%pages);
+ @some_pages = @some_pages[51..$#some_pages];
+ }
+ }
+ if (@tracked_categories) {
+ $user_defined = 1;
+ foreach my $category (@tracked_categories) {
+ if (index($category, ':') < 0) {
+ # Mediawiki requires the Category
+ # prefix, but let's not force the user
+ # to specify it.
+ $category = "Category:" . $category;
+ }
+ my $mw_pages = $mediawiki->list ( {
+ action => 'query',
+ list => 'categorymembers',
+ cmtitle => $category,
+ cmlimit => 'max' } )
+ || die $mediawiki->{error}->{code} . ': ' . $mediawiki->{error}->{details};
+ foreach my $page (@{$mw_pages}) {
+ $pages{$page->{title}} = $page;
+ }
+ }
+ }
+ if (!$user_defined) {
+ # No user-provided list, get the list of pages from
+ # the API.
+ my $mw_pages = $mediawiki->list({
+ action => 'query',
+ list => 'allpages',
+ aplimit => 500,
+ });
+ if (!defined($mw_pages)) {
+ print STDERR "fatal: could not get the list of wiki pages.\n";
+ print STDERR "fatal: '$url' does not appear to be a mediawiki\n";
+ print STDERR "fatal: make sure '$url/api.php' is a valid page.\n";
+ exit 1;
+ }
+ foreach my $page (@{$mw_pages}) {
+ $pages{$page->{title}} = $page;
+ }
+ }
+ return values(%pages);
+}
+
+sub run_git {
+ open(my $git, "-|:encoding(UTF-8)", "git " . $_[0]);
+ my $res = do { local $/; <$git> };
+ close($git);
+
+ return $res;
+}
+
+
+sub get_last_local_revision {
+ # Get note regarding last mediawiki revision
+ my $note = run_git("notes --ref=$remotename/mediawiki show refs/mediawiki/$remotename/master 2>/dev/null");
+ my @note_info = split(/ /, $note);
+
+ my $lastrevision_number;
+ if (!(defined($note_info[0]) && $note_info[0] eq "mediawiki_revision:")) {
+ print STDERR "No previous mediawiki revision found";
+ $lastrevision_number = 0;
+ } else {
+ # Notes are formatted : mediawiki_revision: #number
+ $lastrevision_number = $note_info[1];
+ chomp($lastrevision_number);
+ print STDERR "Last local mediawiki revision found is $lastrevision_number";
+ }
+ return $lastrevision_number;
+}
+
+sub get_last_remote_revision {
+ mw_connect_maybe();
+
+ my @pages = get_mw_pages();
+
+ my $max_rev_num = 0;
+
+ foreach my $page (@pages) {
+ my $id = $page->{pageid};
+
+ my $query = {
+ action => 'query',
+ prop => 'revisions',
+ rvprop => 'ids',
+ pageids => $id,
+ };
+
+ my $result = $mediawiki->api($query);
+
+ my $lastrev = pop(@{$result->{query}->{pages}->{$id}->{revisions}});
+
+ $max_rev_num = ($lastrev->{revid} > $max_rev_num ? $lastrev->{revid} : $max_rev_num);
+ }
+
+ print STDERR "Last remote revision found is $max_rev_num.\n";
+ return $max_rev_num;
+}
+
+# Clean content before sending it to MediaWiki
+sub mediawiki_clean {
+ my $string = shift;
+ my $page_created = shift;
+ # Mediawiki does not allow blank space at the end of a page and ends with a single \n.
+ # This function right trims a string and adds a \n at the end to follow this rule
+ $string =~ s/\s+$//;
+ if ($string eq "" && $page_created) {
+ # Creating empty pages is forbidden.
+ $string = EMPTY_CONTENT;
+ }
+ return $string."\n";
+}
+
+# Filter applied on MediaWiki data before adding them to Git
+sub mediawiki_smudge {
+ my $string = shift;
+ if ($string eq EMPTY_CONTENT) {
+ $string = "";
+ }
+ # This \n is important. This is due to mediawiki's way to handle end of files.
+ return $string."\n";
+}
+
+sub mediawiki_clean_filename {
+ my $filename = shift;
+ $filename =~ s/@{[SLASH_REPLACEMENT]}/\//g;
+ # [, ], |, {, and } are forbidden by MediaWiki, even URL-encoded.
+ # Do a variant of URL-encoding, i.e. looks like URL-encoding,
+ # but with _ added to prevent MediaWiki from thinking this is
+ # an actual special character.
+ $filename =~ s/[\[\]\{\}\|]/sprintf("_%%_%x", ord($&))/ge;
+ # If we use the uri escape before
+ # we should unescape here, before anything
+
+ return $filename;
+}
+
+sub mediawiki_smudge_filename {
+ my $filename = shift;
+ $filename =~ s/\//@{[SLASH_REPLACEMENT]}/g;
+ $filename =~ s/ /_/g;
+ # Decode forbidden characters encoded in mediawiki_clean_filename
+ $filename =~ s/_%_([0-9a-fA-F][0-9a-fA-F])/sprintf("%c", hex($1))/ge;
+ return $filename;
+}
+
+sub literal_data {
+ my ($content) = @_;
+ print STDOUT "data ", bytes::length($content), "\n", $content;
+}
+
+sub mw_capabilities {
+ # Revisions are imported to the private namespace
+ # refs/mediawiki/$remotename/ by the helper and fetched into
+ # refs/remotes/$remotename later by fetch.
+ print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";
+ print STDOUT "import\n";
+ print STDOUT "list\n";
+ print STDOUT "push\n";
+ print STDOUT "\n";
+}
+
+sub mw_list {
+ # MediaWiki do not have branches, we consider one branch arbitrarily
+ # called master, and HEAD pointing to it.
+ print STDOUT "? refs/heads/master\n";
+ print STDOUT "\@refs/heads/master HEAD\n";
+ print STDOUT "\n";
+}
+
+sub mw_option {
+ print STDERR "remote-helper command 'option $_[0]' not yet implemented\n";
+ print STDOUT "unsupported\n";
+}
+
+sub fetch_mw_revisions_for_page {
+ my $page = shift;
+ my $id = shift;
+ my $fetch_from = shift;
+ my @page_revs = ();
+ my $query = {
+ action => 'query',
+ prop => 'revisions',
+ rvprop => 'ids',
+ rvdir => 'newer',
+ rvstartid => $fetch_from,
+ rvlimit => 500,
+ pageids => $id,
+ };
+
+ my $revnum = 0;
+ # Get 500 revisions at a time due to the mediawiki api limit
+ while (1) {
+ my $result = $mediawiki->api($query);
+
+ # Parse each of those 500 revisions
+ foreach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {
+ my $page_rev_ids;
+ $page_rev_ids->{pageid} = $page->{pageid};
+ $page_rev_ids->{revid} = $revision->{revid};
+ push (@page_revs, $page_rev_ids);
+ $revnum++;
+ }
+ last unless $result->{'query-continue'};
+ $query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};
+ }
+ if ($shallow_import && @page_revs) {
+ print STDERR " Found 1 revision (shallow import).\n";
+ @page_revs = sort {$b->{revid} <=> $a->{revid}} (@page_revs);
+ return $page_revs[0];
+ }
+ print STDERR " Found ", $revnum, " revision(s).\n";
+ return @page_revs;
+}
+
+sub fetch_mw_revisions {
+ my $pages = shift; my @pages = @{$pages};
+ my $fetch_from = shift;
+
+ my @revisions = ();
+ my $n = 1;
+ foreach my $page (@pages) {
+ my $id = $page->{pageid};
+
+ print STDERR "page $n/", scalar(@pages), ": ". $page->{title} ."\n";
+ $n++;
+ my @page_revs = fetch_mw_revisions_for_page($page, $id, $fetch_from);
+ @revisions = (@page_revs, @revisions);
+ }
+
+ return ($n, @revisions);
+}
+
+sub import_file_revision {
+ my $commit = shift;
+ my %commit = %{$commit};
+ my $full_import = shift;
+ my $n = shift;
+
+ my $title = $commit{title};
+ my $comment = $commit{comment};
+ my $content = $commit{content};
+ my $author = $commit{author};
+ my $date = $commit{date};
+
+ print STDOUT "commit refs/mediawiki/$remotename/master\n";
+ print STDOUT "mark :$n\n";
+ print STDOUT "committer $author <$author\@$wiki_name> ", $date->epoch, " +0000\n";
+ literal_data($comment);
+
+ # If it's not a clone, we need to know where to start from
+ if (!$full_import && $n == 1) {
+ print STDOUT "from refs/mediawiki/$remotename/master^0\n";
+ }
+ if ($content ne DELETED_CONTENT) {
+ print STDOUT "M 644 inline $title.mw\n";
+ literal_data($content);
+ print STDOUT "\n\n";
+ } else {
+ print STDOUT "D $title.mw\n";
+ }
+
+ # mediawiki revision number in the git note
+ if ($full_import && $n == 1) {
+ print STDOUT "reset refs/notes/$remotename/mediawiki\n";
+ }
+ print STDOUT "commit refs/notes/$remotename/mediawiki\n";
+ print STDOUT "committer $author <$author\@$wiki_name> ", $date->epoch, " +0000\n";
+ literal_data("note added by git-mediawiki");
+ if (!$full_import && $n == 1) {
+ print STDOUT "from refs/notes/$remotename/mediawiki^0\n";
+ }
+ print STDOUT "N inline :$n\n";
+ literal_data("mediawiki_revision: " . $commit{mw_revision});
+ print STDOUT "\n\n";
+}
+
+sub mw_import {
+ $import_started = 1;
+ my $ref = shift;
+ # the remote helper will call "import HEAD" and
+ # "import refs/heads/master"
+ # Since HEAD is a symbolic ref to master (by convention,
+ # followed by the output of the command "list" that we gave),
+ # we don't need to do anything in this case.
+ if ($ref eq "HEAD") {
+ return;
+ }
+
+ mw_connect_maybe();
+
+ my @pages = get_mw_pages();
+
+ print STDERR "Searching revisions...\n";
+ my $last_local = get_last_local_revision();
+ my $fetch_from = $last_local + 1;
+ if ($fetch_from == 1) {
+ print STDERR ", fetching from beginning.\n";
+ } else {
+ print STDERR ", fetching from here.\n";
+ }
+ my ($n, @revisions) = fetch_mw_revisions(\@pages, $fetch_from);
+
+ # Creation of the fast-import stream
+ print STDERR "Fetching & writing export data...\n";
+
+ $n = 0;
+ my $last_timestamp = 0; # Placeholer in case $rev->timestamp is undefined
+
+ foreach my $pagerevid (sort {$a->{revid} <=> $b->{revid}} @revisions) {
+ # fetch the content of the pages
+ my $query = {
+ action => 'query',
+ prop => 'revisions',
+ rvprop => 'content|timestamp|comment|user|ids',
+ revids => $pagerevid->{revid},
+ };
+
+ my $result = $mediawiki->api($query);
+
+ my $rev = pop(@{$result->{query}->{pages}->{$pagerevid->{pageid}}->{revisions}});
+
+ $n++;
+
+ my %commit;
+ $commit{author} = $rev->{user} || 'Anonymous';
+ $commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';
+ $commit{title} = mediawiki_smudge_filename(
+ $result->{query}->{pages}->{$pagerevid->{pageid}}->{title}
+ );
+ $commit{mw_revision} = $pagerevid->{revid};
+ $commit{content} = mediawiki_smudge($rev->{'*'});
+
+ if (!defined($rev->{timestamp})) {
+ $last_timestamp++;
+ } else {
+ $last_timestamp = $rev->{timestamp};
+ }
+ $commit{date} = DateTime::Format::ISO8601->parse_datetime($last_timestamp);
+
+ print STDERR "$n/", scalar(@revisions), ": Revision #$pagerevid->{revid} of $commit{title}\n";
+
+ import_file_revision(\%commit, ($fetch_from == 1), $n);
+ }
+
+ if ($fetch_from == 1 && $n == 0) {
+ print STDERR "You appear to have cloned an empty MediaWiki.\n";
+ # Something has to be done remote-helper side. If nothing is done, an error is
+ # thrown saying that HEAD is refering to unknown object 0000000000000000000
+ # and the clone fails.
+ }
+}
+
+sub error_non_fast_forward {
+ # Native git-push would show this after the summary.
+ # We can't ask it to display it cleanly, so print it
+ # ourselves before.
+ print STDERR "To prevent you from losing history, non-fast-forward updates were rejected\n";
+ print STDERR "Merge the remote changes (e.g. 'git pull') before pushing again. See the\n";
+ print STDERR "'Note about fast-forwards' section of 'git push --help' for details.\n";
+
+ print STDOUT "error $_[0] \"non-fast-forward\"\n";
+ print STDOUT "\n";
+}
+
+sub mw_push_file {
+ my $diff_info = shift;
+ # $diff_info contains a string in this format:
+ # 100644 100644 <sha1_of_blob_before_commit> <sha1_of_blob_now> <status>
+ my @diff_info_split = split(/[ \t]/, $diff_info);
+
+ # Filename, including .mw extension
+ my $complete_file_name = shift;
+ # Commit message
+ my $summary = shift;
+
+ my $new_sha1 = $diff_info_split[3];
+ my $old_sha1 = $diff_info_split[2];
+ my $page_created = ($old_sha1 eq NULL_SHA1);
+ my $page_deleted = ($new_sha1 eq NULL_SHA1);
+ $complete_file_name = mediawiki_clean_filename($complete_file_name);
+
+ if (substr($complete_file_name,-3) eq ".mw"){
+ my $title = substr($complete_file_name,0,-3);
+
+ my $file_content;
+ if ($page_deleted) {
+ # Deleting a page usually requires
+ # special priviledges. A common
+ # convention is to replace the page
+ # with this content instead:
+ $file_content = DELETED_CONTENT;
+ } else {
+ $file_content = run_git("cat-file -p $new_sha1");
+ }
+
+ mw_connect_maybe();
+
+ my $result = $mediawiki->edit( {
+ action => 'edit',
+ summary => $summary,
+ title => $title,
+ text => mediawiki_clean($file_content, $page_created),
+ }, {
+ skip_encoding => 1 # Helps with names with accentuated characters
+ }) || die 'Fatal: Error ' .
+ $mediawiki->{error}->{code} .
+ ' from mediwiki: ' . $mediawiki->{error}->{details};
+ print STDERR "Pushed file : $new_sha1 - $title\n";
+ } else {
+ print STDERR "$complete_file_name not a mediawiki file (Not pushable on this version).\n"
+ }
+}
+
+sub mw_push {
+ my $last_local_revid = get_last_local_revision();
+ print STDERR ".\n"; # Finish sentence started by get_last_local_revision()
+ my $last_remote_revid = get_last_remote_revision();
+
+ # Get sha1 of commit pointed by local HEAD
+ my $HEAD_sha1 = run_git("rev-parse $_[0] 2>/dev/null"); chomp($HEAD_sha1);
+ # Get sha1 of commit pointed by remotes/$remotename/master
+ my $remoteorigin_sha1 = run_git("rev-parse refs/remotes/$remotename/master 2>/dev/null");
+ chomp($remoteorigin_sha1);
+
+ if ($last_local_revid > 0 &&
+ $last_local_revid < $last_remote_revid){
+ return error_non_fast_forward($_[0]);
+ }
+
+ if ($HEAD_sha1 eq $remoteorigin_sha1) {
+ print STDOUT "\n";
+ return;
+ }
+
+ # Get every commit in between HEAD and refs/remotes/origin/master,
+ # including HEAD and refs/remotes/origin/master
+ my @commit_pairs = ();
+ if ($last_local_revid > 0) {
+ my $parsed_sha1 = $remoteorigin_sha1;
+ # Find a path from last MediaWiki commit to pushed commit
+ while ($parsed_sha1 ne $HEAD_sha1) {
+ my @commit_info = grep(/^$parsed_sha1/, split(/\n/, run_git("rev-list --children $_[0]")));
+ if (!@commit_info) {
+ return error_non_fast_forward($_[0]);
+ }
+ my @commit_info_split = split(/ |\n/, $commit_info[0]);
+ # $commit_info_split[1] is the sha1 of the commit to export
+ # $commit_info_split[0] is the sha1 of its direct child
+ push (@commit_pairs, \@commit_info_split);
+ $parsed_sha1 = $commit_info_split[1];
+ }
+ } else {
+ # No remote mediawiki revision. Export the whole
+ # history (linearized with --first-parent)
+ print STDERR "Warning: no common ancestor, pushing complete history\n";
+ my $history = run_git("rev-list --first-parent --children $_[0]");
+ my @history = split('\n', $history);
+ @history = @history[1..$#history];
+ foreach my $line (reverse @history) {
+ my @commit_info_split = split(/ |\n/, $line);
+ push (@commit_pairs, \@commit_info_split);
+ }
+ }
+
+ foreach my $commit_info_split (@commit_pairs) {
+ my $sha1_child = @{$commit_info_split}[0];
+ my $sha1_commit = @{$commit_info_split}[1];
+ my $diff_infos = run_git("diff-tree -r --raw -z $sha1_child $sha1_commit");
+ # TODO: we could detect rename, and encode them with a #redirect on the wiki.
+ # TODO: for now, it's just a delete+add
+ my @diff_info_list = split(/\0/, $diff_infos);
+ # Keep the first line of the commit message as mediawiki comment for the revision
+ my $commit_msg = (split(/\n/, run_git("show --pretty=format:\"%s\" $sha1_commit")))[0];
+ chomp($commit_msg);
+ # Push every blob
+ while (@diff_info_list) {
+ # git diff-tree -z gives an output like
+ # <metadata>\0<filename1>\0
+ # <metadata>\0<filename2>\0
+ # and we've split on \0.
+ my $info = shift(@diff_info_list);
+ my $file = shift(@diff_info_list);
+ mw_push_file($info, $file, $commit_msg);
+ }
+ }
+
+ print STDOUT "ok $_[1]\n";
+ print STDOUT "\n";
+
+ print STDERR "Just pushed some revisions to MediaWiki.\n";
+ print STDERR "The pushed revisions now have to be re-imported, and your current branch\n";
+ print STDERR "needs to be updated with these re-imported commits. You can do this with\n";
+ print STDERR "\n";
+ print STDERR " git pull --rebase\n";
+ print STDERR "\n";
+}
diff --git a/contrib/mw-to-git/git-remote-mediawiki.txt b/contrib/mw-to-git/git-remote-mediawiki.txt
new file mode 100644
index 0000000..4d211f5
--- /dev/null
+++ b/contrib/mw-to-git/git-remote-mediawiki.txt
@@ -0,0 +1,7 @@
+Git-Mediawiki is a project which aims the creation of a gate
+between git and mediawiki, allowing git users to push and pull
+objects from mediawiki just as one would do with a classic git
+repository thanks to remote-helpers.
+
+For more information, visit the wiki at
+https://github.com/Bibzball/Git-Mediawiki/wiki
--
1.7.6.585.g5929f.dirty
^ permalink raw reply related
* Re: [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)
From: Junio C Hamano @ 2011-08-26 17:53 UTC (permalink / raw)
To: Matthieu Moy
Cc: git, Jeremie Nikaes, Arnaud Lacurie, Claire Fousse, David Amouyal,
Matthieu Moy, Sylvain Boulmé
In-Reply-To: <1314378689-8997-2-git-send-email-Matthieu.Moy@imag.fr>
Matthieu Moy <Matthieu.Moy@imag.fr> writes:
> In short, the changes since v3 are:
>
> * Adapt to newer Git, which seem to require a "done" command at the
> end of the fast-import stream. I don't understand why this is
> needed, since fast-import is called without the --done flag by
> remote-helpers, but if I don't do this, "git fetch" doesn't
> terminate and keeps waiting ...
Hmmmm, is this a regression in fast-import? Can this be bisected if so?
^ permalink raw reply
* Re: [PATCH 1/2] fast-import: initialize variable require_explicit_termination
From: Junio C Hamano @ 2011-08-26 17:51 UTC (permalink / raw)
To: Matthieu Moy; +Cc: git
In-Reply-To: <1314378689-8997-1-git-send-email-Matthieu.Moy@imag.fr>
Matthieu Moy <Matthieu.Moy@imag.fr> writes:
> The uninitialized variable seems harmless in practice, but let's still be clean.
It is not "in practice", but by definition, file scope "static int"
variables are initialized to 0 by the C language (a typical implementation
achieves this by placing the variable in BSS section).
Please do not write unnecessary " = 0" there.
>
> Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
> ---
> For some reason, remote helpers seem to be forced to use the "done"
> command now. Investing why, I found this, but that wasn't what I was
> looking for.
>
> fast-import.c | 2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
>
> diff --git a/fast-import.c b/fast-import.c
> index 7cc2262..ed8f3cd 100644
> --- a/fast-import.c
> +++ b/fast-import.c
> @@ -355,7 +355,7 @@ static unsigned int cmd_save = 100;
> static uintmax_t next_mark;
> static struct strbuf new_data = STRBUF_INIT;
> static int seen_data_command;
> -static int require_explicit_termination;
> +static int require_explicit_termination = 0;
>
> /* Signal handling */
> static volatile sig_atomic_t checkpoint_requested;
^ permalink raw reply
* Re: [PATCHv2 4/5] branch: introduce --list option
From: Junio C Hamano @ 2011-08-26 17:43 UTC (permalink / raw)
To: Michael J Gruber; +Cc: git, Jeff King
In-Reply-To: <0785cac235c3b45537cf161c86dde8e798c4ff3e.1314367414.git.git@drmicha.warpmail.net>
Michael J Gruber <git@drmicha.warpmail.net> writes:
> Currently, there is no way to invoke the list mode explicitly.
..., without giving -v to force verbose output.
> Introduce a --list option which invokes the list mode. This will be
> beneficial for invoking list mode with pattern matching, which otherwise
> would be interpreted as branch creation.
>
> Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
> ---
> @@ -20,7 +20,8 @@ DESCRIPTION
>
> With no arguments, existing branches are listed and the current branch will
> be highlighted with an asterisk. Option `-r` causes the remote-tracking
> -branches to be listed, and option `-a` shows both.
> +branches to be listed, and option `-a` shows both. This list mode is also
> +activated by the `--list` and `-v` options (see below).
Very good to mention "and -v" here ;-)
> diff --git a/t/t3203-branch-output.sh b/t/t3203-branch-output.sh
> index 6b7c118..61e095c 100755
> --- a/t/t3203-branch-output.sh
> +++ b/t/t3203-branch-output.sh
> @@ -32,6 +32,20 @@ test_expect_success 'git branch shows local branches' '
> test_cmp expect actual
> '
>
> +test_expect_success 'git branch --list shows local branches' '
> + git branch --list >actual &&
> + test_cmp expect actual
> +'
> +
> +cat >expect <<'EOF'
> + branch-one
> + branch-two
> +EOF
> +test_expect_success 'git branch --list pattern shows matching local branches' '
> + git branch --list branch* >actual &&
> + test_cmp expect actual
> +'
> +
Could we have a test to check the code you updated to sanity check the
combination of options as well? I suspect the reason your initial round
botched the "branch -v -m foo" without realizing may be because we do not
cover the error checking.
^ permalink raw reply
* Re: [PATCHv2 3/5] git-branch: introduce missing long forms for the options
From: Junio C Hamano @ 2011-08-26 17:13 UTC (permalink / raw)
To: Michael J Gruber; +Cc: git, Jeff King
In-Reply-To: <c79179fa3476629ce47556c219719495c213f5f9.1314367414.git.git@drmicha.warpmail.net>
Michael J Gruber <git@drmicha.warpmail.net> writes:
> @@ -100,9 +103,11 @@ OPTIONS
> Same as `--color=never`.
>
> -r::
> +--remotes::
> List or delete (if used with -d) the remote-tracking branches.
I am not sure if this should be "--remoteS".
^ permalink raw reply
* [PATCH 2/2] Add a remote helper to interact with mediawiki (fetch & push)
From: Matthieu Moy @ 2011-08-26 17:11 UTC (permalink / raw)
To: git, gitster
Cc: Jeremie Nikaes, Arnaud Lacurie, Claire Fousse, David Amouyal,
Matthieu Moy, Sylvain Boulmé, Matthieu Moy
In-Reply-To: <1314378689-8997-1-git-send-email-Matthieu.Moy@imag.fr>
From: Jeremie Nikaes <jeremie.nikaes@ensimag.imag.fr>
Implement a gate between git and mediawiki, allowing git users to push
and pull objects from mediawiki just as one would do with a classic git
repository thanks to remote-helpers.
The following packages need to be installed (available on common
repositories):
libmediawiki-api-perl
libdatetime-format-iso8601-perl
Use remote helpers in order to be as transparent as possible to the git
user.
Download Mediawiki revisions through the Mediawiki API and then
fast-import into git.
Mediawiki revision number and git commits are linked thanks to notes
bound to commits.
The import part is done on a refs/mediawiki/<remote> branch before
coming to refs/remote/origin/master (Huge thanks to Jonathan Nieder
for his help)
We use UTF-8 everywhere: use encoding 'utf8'; does most of the job, but
we also read the output of Git commands in UTF-8 with the small helper
run_git, and write to the console (STDERR) in UTF-8. This allows a
seamless use of non-ascii characters in page titles, but hasn't been
tested on non-UTF-8 systems. In particular, UTF-8 encoding for filenames
could raise problems if different file systems handle UTF-8 filenames
differently. A uri_escape of mediawiki filenames could be imaginable, and
is still to be discussed further.
Partial cloning is supported using one of:
git clone -c remote.origin.pages='A_Page Another_Page' mediawiki::http://wikiurl
git clone -c remote.origin.categories='Some_Category' mediawiki::http://wikiurl
git clone -c remote.origin.shallow='True' mediawiki::http://wikiurl
Thanks to notes metadata, it is possible to compare remote and local last
mediawiki revision to warn non-fast forward pushes and "everything
up-to-date" case.
When allowed, push looks for each commit between remotes/origin/master
and HEAD, catches every blob related to these commit and push them in
chronological order. To do so, it uses git rev-list --children HEAD and
travels the tree from remotes/origin/master to HEAD through children. In
other words :
* Shortest path from remotes/origin/master to HEAD
* For each commit encountered, push blobs related to this commit
Signed-off-by: Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>
Signed-off-by: Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>
Signed-off-by: Claire Fousse <claire.fousse@ensimag.imag.fr>
Signed-off-by: David Amouyal <david.amouyal@ensimag.imag.fr>
Signed-off-by: Matthieu Moy <matthieu.moy@grenoble-inp.fr>
Signed-off-by: Sylvain Boulmé <sylvain.boulme@imag.fr>
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
This is a resurection of an old patch serie.
The previous round was here:
http://thread.gmane.org/gmane.comp.version-control.git/175523
Maintaining two separate patches for import/export was getting
painful, so I've merged them into a single patch.
Git-Mediawiki is originally a student project, and since the students
didn't have time to finish the work this summer, I've taken time to
do some cleanup and testing.
In short, the changes since v3 are:
* Adapt to newer Git, which seem to require a "done" command at the
end of the fast-import stream. I don't understand why this is
needed, since fast-import is called without the --done flag by
remote-helpers, but if I don't do this, "git fetch" doesn't
terminate and keeps waiting ...
* Allow importing just a category
* Support shallow clone and fetch (for fetch, this means fetch just
the last revision each time).
* Basic support for authentication (with password cleartext in
.git/config :-( ).
* Support pushing to a new wiki (i.e. not the one you've cloned to).
* Accented characters in filenames now displayed correctly on the
terminal.
* Support page deletion (by replacing the page content with
[[Category:Deleted]]).
* Support creation of empty files
* Support forbidden characters {}[]| in filenames
* Support partial clone with more than 50 pages
I'd like to get this merged in contrib/, so that the code be in a
safe place, where other people can take care of it too. I'll probably
offer a student project "improve Git-Mediawiki" to my students next
June.
contrib/mw-to-git/git-remote-mediawiki | 722 ++++++++++++++++++++++++++++
contrib/mw-to-git/git-remote-mediawiki.txt | 7 +
2 files changed, 729 insertions(+), 0 deletions(-)
create mode 100755 contrib/mw-to-git/git-remote-mediawiki
create mode 100644 contrib/mw-to-git/git-remote-mediawiki.txt
diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
new file mode 100755
index 0000000..62c6794
--- /dev/null
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -0,0 +1,722 @@
+#! /usr/bin/perl
+
+# Copyright (C) 2011
+# Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>
+# Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>
+# Claire Fousse <claire.fousse@ensimag.imag.fr>
+# David Amouyal <david.amouyal@ensimag.imag.fr>
+# Matthieu Moy <matthieu.moy@grenoble-inp.fr>
+# License: GPL v2 or later
+
+# Gateway between Git and MediaWiki.
+# https://github.com/Bibzball/Git-Mediawiki/wiki
+#
+# Known limitations:
+#
+# - Only wiki pages are managed, no support for [[File:...]]
+# attachments.
+#
+# - Poor performance in the best case: it takes forever to check
+# whether we're up-to-date (on fetch or push) or to fetch a few
+# revisions from a large wiki, because we use exclusively a
+# page-based synchronization. We could switch to a wiki-wide
+# synchronization when the synchronization involves few revisions
+# but the wiki is large.
+#
+# - Git renames could be turned into MediaWiki renames (see TODO
+# below)
+#
+# - login/password support requires the user to write the password
+# cleartext in a file (see TODO below).
+#
+# - No way to import "one page, and all pages included in it"
+#
+# - Multiple remote MediaWikis have not been very well tested.
+
+use strict;
+use MediaWiki::API;
+use DateTime::Format::ISO8601;
+use encoding 'utf8';
+
+# use encoding 'utf8' doesn't change STDERROR
+# but we're going to output UTF-8 filenames to STDERR
+binmode STDERR, ":utf8";
+
+use URI::Escape;
+use warnings;
+
+# Mediawiki filenames can contain forward slashes. This variable decides by which pattern they should be replaced
+use constant SLASH_REPLACEMENT => "%2F";
+
+# It's not always possible to delete pages (may require some
+# priviledges). Deleted pages are replaced with this content.
+use constant DELETED_CONTENT => "[[Category:Deleted]]\n";
+
+# It's not possible to create empty pages. New empty files in Git are
+# sent with this content instead.
+use constant EMPTY_CONTENT => "<!-- empty page -->\n";
+
+# used to reflect file creation or deletion in diff.
+use constant NULL_SHA1 => "0000000000000000000000000000000000000000";
+
+my $remotename = $ARGV[0];
+my $url = $ARGV[1];
+
+# Accept both space-separated and multiple keys in config file.
+# Spaces should be written as _ anyway because we'll use chomp.
+my @tracked_pages = split(/[ \n]/, run_git("config --get-all remote.". $remotename .".pages"));
+chomp(@tracked_pages);
+
+# Just like @tracked_pages, but for MediaWiki categories.
+my @tracked_categories = split(/[ \n]/, run_git("config --get-all remote.". $remotename .".categories"));
+chomp(@tracked_categories);
+
+my $wiki_login = run_git("config --get remote.". $remotename .".mwLogin");
+# TODO: ideally, this should be able to read from keyboard, but we're
+# inside a remote helper, so our stdin is connect to git, not to a
+# terminal.
+my $wiki_passwd = run_git("config --get remote.". $remotename .".mwPassword");
+chomp ($wiki_login);
+chomp ($wiki_passwd);
+
+# Import only last revisions (both for clone and fetch)
+my $shallow_import = run_git("config --get --bool remote.". $remotename .".shallow");
+chomp($shallow_import);
+$shallow_import = ($shallow_import eq "true");
+
+my $wiki_name = $url;
+$wiki_name =~ s/[^\/]*:\/\///;
+
+my $import_started;
+
+# Commands parser
+my $entry;
+my @cmd;
+while (<STDIN>) {
+ chomp;
+ @cmd = split(/ /);
+ if (defined($cmd[0])) {
+ # Line not blank
+ if ($cmd[0] eq "capabilities") {
+ die("Too many arguments for capabilities") unless (!defined($cmd[1]));
+ mw_capabilities();
+ } elsif ($cmd[0] eq "list") {
+ die("Too many arguments for list") unless (!defined($cmd[2]));
+ mw_list($cmd[1]);
+ } elsif ($cmd[0] eq "import") {
+ die("Invalid arguments for import") unless ($cmd[1] ne "" && !defined($cmd[2]));
+ mw_import($cmd[1]);
+ } elsif ($cmd[0] eq "option") {
+ die("Too many arguments for option") unless ($cmd[1] ne "" && $cmd[2] ne "" && !defined($cmd[3]));
+ mw_option($cmd[1],$cmd[2]);
+ } elsif ($cmd[0] eq "push") {
+ # Check the pattern <src>:<dst>
+ my @pushargs = split(/:/,$cmd[1]);
+ die("Invalid arguments for push") unless ($pushargs[1] ne "" && !defined($pushargs[2]));
+ mw_push($pushargs[0],$pushargs[1]);
+ } else {
+ print STDERR "Unknown command. Aborting...\n";
+ last;
+ }
+ } else {
+ # blank line: we should terminate
+ last;
+ }
+
+ BEGIN { $| = 1 } # flush STDOUT, to make sure the previous
+ # command is fully processed.
+}
+# End of input
+if ($import_started) {
+ # Terminate the fast-import stream properly.
+ # Git requires one "done" command, and only
+ # one This is OK since we only have one
+ # branch, so import will be called only once
+ # (plus once for HEAD, for which we won't
+ # reach this point).
+ print STDOUT "done\n";
+}
+BEGIN { $| = 1 };
+if (!eof(STDIN)) {
+ # Wait for Git to terminate. If we don't, git fetch
+ # (transport-helper.c's sendline function) will try to write
+ # to our stdin, which will be closed, and git fetch will be
+ # killed. That's probably a bug in transport-helper.c, but in
+ # the meantime ...
+ sleep .1;
+};
+
+########################## Functions ##############################
+
+# MediaWiki API instance, created lazily.
+my $mediawiki;
+
+sub mw_connect_maybe {
+ if ($mediawiki) {
+ return;
+ }
+ $mediawiki = MediaWiki::API->new;
+ $mediawiki->{config}->{api_url} = "$url/api.php";
+ if ($wiki_login) {
+ if (!$mediawiki->login({
+ lgname => $wiki_login,
+ lgpassword => $wiki_passwd,
+ })) {
+ print STDERR "Failed to log in mediawiki user \"$wiki_login\" on $url\n";
+ print STDERR "(error " .
+ $mediawiki->{error}->{code} . ': ' .
+ $mediawiki->{error}->{details} . ")\n";
+ exit 1;
+ } else {
+ print STDERR "Logged in with user \"$wiki_login\".\n";
+ }
+ }
+}
+
+sub get_mw_first_pages {
+ my $some_pages = shift;
+ my @some_pages = @{$some_pages};
+
+ my $pages = shift;
+
+ # pattern 'page1|page2|...' required by the API
+ my $titles = join('|', @some_pages);
+
+ my $mw_pages = $mediawiki->api({
+ action => 'query',
+ titles => $titles,
+ });
+ if (!defined($mw_pages)) {
+ print STDERR "fatal: could not query the list of wiki pages.\n";
+ print STDERR "fatal: '$url' does not appear to be a mediawiki\n";
+ print STDERR "fatal: make sure '$url/api.php' is a valid page.\n";
+ exit 1;
+ }
+ while (my ($id, $page) = each (%{$mw_pages->{query}->{pages}})) {
+ if ($id < 0) {
+ print STDERR "Warning: page $page->{title} not found on wiki\n";
+ } else {
+ $pages->{$page->{title}} = $page;
+ }
+ }
+}
+
+sub get_mw_pages {
+ mw_connect_maybe();
+
+ my %pages; # hash on page titles to avoid duplicates
+ my $user_defined;
+ if (@tracked_pages) {
+ $user_defined = 1;
+ # The user provided a list of pages titles, but we
+ # still need to query the API to get the page IDs.
+
+ my @some_pages = @tracked_pages;
+ while (@some_pages) {
+ my $last = 50;
+ if ($#some_pages < $last) {
+ $last = $#some_pages;
+ }
+ my @slice = @some_pages[0..$last];
+ get_mw_first_pages(\@slice, \%pages);
+ @some_pages = @some_pages[51..$#some_pages];
+ }
+ }
+ if (@tracked_categories) {
+ $user_defined = 1;
+ foreach my $category (@tracked_categories) {
+ if (index($category, ':') < 0) {
+ # Mediawiki requires the Category
+ # prefix, but let's not force the user
+ # to specify it.
+ $category = "Category:" . $category;
+ }
+ my $mw_pages = $mediawiki->list ( {
+ action => 'query',
+ list => 'categorymembers',
+ cmtitle => $category,
+ cmlimit => 'max' } )
+ || die $mediawiki->{error}->{code} . ': ' . $mediawiki->{error}->{details};
+ foreach my $page (@{$mw_pages}) {
+ $pages{$page->{title}} = $page;
+ }
+ }
+ }
+ if (!$user_defined) {
+ # No user-provided list, get the list of pages from
+ # the API.
+ my $mw_pages = $mediawiki->list({
+ action => 'query',
+ list => 'allpages',
+ aplimit => 500,
+ });
+ if (!defined($mw_pages)) {
+ print STDERR "fatal: could not get the list of wiki pages.\n";
+ print STDERR "fatal: '$url' does not appear to be a mediawiki\n";
+ print STDERR "fatal: make sure '$url/api.php' is a valid page.\n";
+ exit 1;
+ }
+ foreach my $page (@{$mw_pages}) {
+ $pages{$page->{title}} = $page;
+ }
+ }
+ return values(%pages);
+}
+
+sub run_git {
+ open(my $git, "-|:encoding(UTF-8)", "git " . $_[0]);
+ my $res = do { local $/; <$git> };
+ close($git);
+
+ return $res;
+}
+
+
+sub get_last_local_revision {
+ # Get note regarding last mediawiki revision
+ my $note = run_git("notes --ref=$remotename/mediawiki show refs/mediawiki/$remotename/master 2>/dev/null");
+ my @note_info = split(/ /, $note);
+
+ my $lastrevision_number;
+ if (!(defined($note_info[0]) && $note_info[0] eq "mediawiki_revision:")) {
+ print STDERR "No previous mediawiki revision found";
+ $lastrevision_number = 0;
+ } else {
+ # Notes are formatted : mediawiki_revision: #number
+ $lastrevision_number = $note_info[1];
+ chomp($lastrevision_number);
+ print STDERR "Last local mediawiki revision found is $lastrevision_number";
+ }
+ return $lastrevision_number;
+}
+
+sub get_last_remote_revision {
+ mw_connect_maybe();
+
+ my @pages = get_mw_pages();
+
+ my $max_rev_num = 0;
+
+ foreach my $page (@pages) {
+ my $id = $page->{pageid};
+
+ my $query = {
+ action => 'query',
+ prop => 'revisions',
+ rvprop => 'ids',
+ pageids => $id,
+ };
+
+ my $result = $mediawiki->api($query);
+
+ my $lastrev = pop(@{$result->{query}->{pages}->{$id}->{revisions}});
+
+ $max_rev_num = ($lastrev->{revid} > $max_rev_num ? $lastrev->{revid} : $max_rev_num);
+ }
+
+ print STDERR "Last remote revision found is $max_rev_num.\n";
+ return $max_rev_num;
+}
+
+# Clean content before sending it to MediaWiki
+sub mediawiki_clean {
+ my $string = shift;
+ my $page_created = shift;
+ # Mediawiki does not allow blank space at the end of a page and ends with a single \n.
+ # This function right trims a string and adds a \n at the end to follow this rule
+ $string =~ s/\s+$//;
+ if ($string eq "" && $page_created) {
+ # Creating empty pages is forbidden.
+ $string = EMPTY_CONTENT;
+ }
+ return $string."\n";
+}
+
+# Filter applied on MediaWiki data before adding them to Git
+sub mediawiki_smudge {
+ my $string = shift;
+ if ($string eq EMPTY_CONTENT) {
+ $string = "";
+ }
+ # This \n is important. This is due to mediawiki's way to handle end of files.
+ return $string."\n";
+}
+
+sub mediawiki_clean_filename {
+ my $filename = shift;
+ $filename =~ s/@{[SLASH_REPLACEMENT]}/\//g;
+ # [, ], |, {, and } are forbidden by MediaWiki, even URL-encoded.
+ # Do a variant of URL-encoding, i.e. looks like URL-encoding,
+ # but with _ added to prevent MediaWiki from thinking this is
+ # an actual special character.
+ $filename =~ s/[\[\]\{\}\|]/sprintf("_%%_%x", ord($&))/ge;
+ # If we use the uri escape before
+ # we should unescape here, before anything
+
+ return $filename;
+}
+
+sub mediawiki_smudge_filename {
+ my $filename = shift;
+ $filename =~ s/\//@{[SLASH_REPLACEMENT]}/g;
+ $filename =~ s/ /_/g;
+ # Decode forbidden characters encoded in mediawiki_clean_filename
+ $filename =~ s/_%_([0-9a-fA-F][0-9a-fA-F])/sprintf("%c", hex($1))/ge;
+ return $filename;
+}
+
+sub literal_data {
+ my ($content) = @_;
+ print STDOUT "data ", bytes::length($content), "\n", $content;
+}
+
+sub mw_capabilities {
+ # Revisions are imported to the private namespace
+ # refs/mediawiki/$remotename/ by the helper and fetched into
+ # refs/remotes/$remotename later by fetch.
+ print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";
+ print STDOUT "import\n";
+ print STDOUT "list\n";
+ print STDOUT "push\n";
+ print STDOUT "\n";
+}
+
+sub mw_list {
+ # MediaWiki do not have branches, we consider one branch arbitrarily
+ # called master, and HEAD pointing to it.
+ print STDOUT "? refs/heads/master\n";
+ print STDOUT "\@refs/heads/master HEAD\n";
+ print STDOUT "\n";
+}
+
+sub mw_option {
+ print STDERR "remote-helper command 'option $_[0]' not yet implemented\n";
+ print STDOUT "unsupported\n";
+}
+
+sub fetch_mw_revisions_for_page {
+ my $page = shift;
+ my $id = shift;
+ my $fetch_from = shift;
+ my @page_revs = ();
+ my $query = {
+ action => 'query',
+ prop => 'revisions',
+ rvprop => 'ids',
+ rvdir => 'newer',
+ rvstartid => $fetch_from,
+ rvlimit => 500,
+ pageids => $id,
+ };
+
+ my $revnum = 0;
+ # Get 500 revisions at a time due to the mediawiki api limit
+ while (1) {
+ my $result = $mediawiki->api($query);
+
+ # Parse each of those 500 revisions
+ foreach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {
+ my $page_rev_ids;
+ $page_rev_ids->{pageid} = $page->{pageid};
+ $page_rev_ids->{revid} = $revision->{revid};
+ push (@page_revs, $page_rev_ids);
+ $revnum++;
+ }
+ last unless $result->{'query-continue'};
+ $query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};
+ }
+ if ($shallow_import && @page_revs) {
+ print STDERR " Found 1 revision (shallow import).\n";
+ @page_revs = sort {$b->{revid} <=> $a->{revid}} (@page_revs);
+ return $page_revs[0];
+ }
+ print STDERR " Found ", $revnum, " revision(s).\n";
+ return @page_revs;
+}
+
+sub fetch_mw_revisions {
+ my $pages = shift; my @pages = @{$pages};
+ my $fetch_from = shift;
+
+ my @revisions = ();
+ my $n = 1;
+ foreach my $page (@pages) {
+ my $id = $page->{pageid};
+
+ print STDERR "page $n/", scalar(@pages), ": ". $page->{title} ."\n";
+ $n++;
+ my @page_revs = fetch_mw_revisions_for_page($page, $id, $fetch_from);
+ @revisions = (@page_revs, @revisions);
+ }
+
+ return ($n, @revisions);
+}
+
+sub import_file_revision {
+ my $commit = shift;
+ my %commit = %{$commit};
+ my $full_import = shift;
+ my $n = shift;
+
+ my $title = $commit{title};
+ my $comment = $commit{comment};
+ my $content = $commit{content};
+ my $author = $commit{author};
+ my $date = $commit{date};
+
+ print STDOUT "commit refs/mediawiki/$remotename/master\n";
+ print STDOUT "mark :$n\n";
+ print STDOUT "committer $author <$author\@$wiki_name> ", $date->epoch, " +0000\n";
+ literal_data($comment);
+
+ # If it's not a clone, we need to know where to start from
+ if (!$full_import && $n == 1) {
+ print STDOUT "from refs/mediawiki/$remotename/master^0\n";
+ }
+ if ($content ne DELETED_CONTENT) {
+ print STDOUT "M 644 inline $title.mw\n";
+ literal_data($content);
+ print STDOUT "\n\n";
+ } else {
+ print STDOUT "D $title.mw\n";
+ }
+
+ # mediawiki revision number in the git note
+ if ($full_import && $n == 1) {
+ print STDOUT "reset refs/notes/$remotename/mediawiki\n";
+ }
+ print STDOUT "commit refs/notes/$remotename/mediawiki\n";
+ print STDOUT "committer $author <$author\@$wiki_name> ", $date->epoch, " +0000\n";
+ literal_data("note added by git-mediawiki");
+ if (!$full_import && $n == 1) {
+ print STDOUT "from refs/notes/$remotename/mediawiki^0\n";
+ }
+ print STDOUT "N inline :$n\n";
+ literal_data("mediawiki_revision: " . $commit{mw_revision});
+ print STDOUT "\n\n";
+}
+
+sub mw_import {
+ $import_started = 1;
+ my $ref = shift;
+ # the remote helper will call "import HEAD" and
+ # "import refs/heads/master"
+ # Since HEAD is a symbolic ref to master (by convention,
+ # followed by the output of the command "list" that we gave),
+ # we don't need to do anything in this case.
+ if ($ref eq "HEAD") {
+ return;
+ }
+
+ mw_connect_maybe();
+
+ my @pages = get_mw_pages();
+
+ print STDERR "Searching revisions...\n";
+ my $last_local = get_last_local_revision();
+ my $fetch_from = $last_local + 1;
+ if ($fetch_from == 1) {
+ print STDERR ", fetching from beginning.\n";
+ } else {
+ print STDERR ", fetching from here.\n";
+ }
+ my ($n, @revisions) = fetch_mw_revisions(\@pages, $fetch_from);
+
+ # Creation of the fast-import stream
+ print STDERR "Fetching & writing export data...\n";
+
+ $n = 0;
+ my $last_timestamp = 0; # Placeholer in case $rev->timestamp is undefined
+
+ foreach my $pagerevid (sort {$a->{revid} <=> $b->{revid}} @revisions) {
+ # fetch the content of the pages
+ my $query = {
+ action => 'query',
+ prop => 'revisions',
+ rvprop => 'content|timestamp|comment|user|ids',
+ revids => $pagerevid->{revid},
+ };
+
+ my $result = $mediawiki->api($query);
+
+ my $rev = pop(@{$result->{query}->{pages}->{$pagerevid->{pageid}}->{revisions}});
+
+ $n++;
+
+ my %commit;
+ $commit{author} = $rev->{user} || 'Anonymous';
+ $commit{comment} = $rev->{comment} || '*Empty MediaWiki Message*';
+ $commit{title} = mediawiki_smudge_filename(
+ $result->{query}->{pages}->{$pagerevid->{pageid}}->{title}
+ );
+ $commit{mw_revision} = $pagerevid->{revid};
+ $commit{content} = mediawiki_smudge($rev->{'*'});
+
+ if (!defined($rev->{timestamp})) {
+ $last_timestamp++;
+ } else {
+ $last_timestamp = $rev->{timestamp};
+ }
+ $commit{date} = DateTime::Format::ISO8601->parse_datetime($last_timestamp);
+
+ print STDERR "$n/", scalar(@revisions), ": Revision #$pagerevid->{revid} of $commit{title}\n";
+
+ import_file_revision(\%commit, ($fetch_from == 1), $n);
+ }
+
+ if ($fetch_from == 1) {
+ if ($n != 0) {
+ print STDOUT "reset $ref\n";
+ print STDOUT "from :$n\n";
+ } else {
+ print STDERR "You appear to have cloned an empty mediawiki\n";
+ # Something has to be done remote-helper side. If nothing is done, an error is
+ # thrown saying that HEAD is refering to unknown object 0000000000000000000
+ }
+ }
+}
+
+sub error_non_fast_forward {
+ # Native git-push would show this after the summary.
+ # We can't ask it to display it cleanly, so print it
+ # ourselves before.
+ print STDERR "To prevent you from losing history, non-fast-forward updates were rejected\n";
+ print STDERR "Merge the remote changes (e.g. 'git pull') before pushing again. See the\n";
+ print STDERR "'Note about fast-forwards' section of 'git push --help' for details.\n";
+
+ print STDOUT "error $_[0] \"non-fast-forward\"\n";
+ print STDOUT "\n";
+}
+
+sub mw_push_file {
+ my $diff_info = shift;
+ # $diff_info contains a string in this format:
+ # 100644 100644 <sha1_of_blob_before_commit> <sha1_of_blob_now> <status>
+ my @diff_info_split = split(/[ \t]/, $diff_info);
+
+ # Filename, including .mw extension
+ my $complete_file_name = shift;
+ # Commit message
+ my $summary = shift;
+
+ my $new_sha1 = $diff_info_split[3];
+ my $old_sha1 = $diff_info_split[2];
+ my $page_created = ($old_sha1 eq NULL_SHA1);
+ my $page_deleted = ($new_sha1 eq NULL_SHA1);
+ $complete_file_name = mediawiki_clean_filename($complete_file_name);
+
+ if (substr($complete_file_name,-3) eq ".mw"){
+ my $title = substr($complete_file_name,0,-3);
+
+ my $file_content;
+ if ($page_deleted) {
+ # Deleting a page usually requires
+ # special priviledges. A common
+ # convention is to replace the page
+ # with this content instead:
+ $file_content = DELETED_CONTENT;
+ } else {
+ $file_content = run_git("cat-file -p $new_sha1");
+ }
+
+ mw_connect_maybe();
+
+ my $result = $mediawiki->edit( {
+ action => 'edit',
+ summary => $summary,
+ title => $title,
+ text => mediawiki_clean($file_content, $page_created),
+ }, {
+ skip_encoding => 1 # Helps with names with accentuated characters
+ }) || die 'Fatal: Error ' .
+ $mediawiki->{error}->{code} .
+ ' from mediwiki: ' . $mediawiki->{error}->{details};
+ print STDERR "Pushed file : $new_sha1 - $title\n";
+ } else {
+ print STDERR "$complete_file_name not a mediawiki file (Not pushable on this version).\n"
+ }
+}
+
+sub mw_push {
+ my $last_local_revid = get_last_local_revision();
+ print STDERR ".\n"; # Finish sentence started by get_last_local_revision()
+ my $last_remote_revid = get_last_remote_revision();
+
+ # Get sha1 of commit pointed by local HEAD
+ my $HEAD_sha1 = run_git("rev-parse $_[0] 2>/dev/null"); chomp($HEAD_sha1);
+ # Get sha1 of commit pointed by remotes/$remotename/master
+ my $remoteorigin_sha1 = run_git("rev-parse refs/remotes/$remotename/master 2>/dev/null");
+ chomp($remoteorigin_sha1);
+
+ if ($last_local_revid > 0 &&
+ $last_local_revid < $last_remote_revid){
+ return error_non_fast_forward($_[0]);
+ }
+
+ if ($HEAD_sha1 eq $remoteorigin_sha1) {
+ print STDOUT "\n";
+ return;
+ }
+
+ # Get every commit in between HEAD and refs/remotes/origin/master,
+ # including HEAD and refs/remotes/origin/master
+ my @commit_pairs = ();
+ if ($last_local_revid > 0) {
+ my $parsed_sha1 = $remoteorigin_sha1;
+ # Find a path from last MediaWiki commit to pushed commit
+ while ($parsed_sha1 ne $HEAD_sha1) {
+ my @commit_info = grep(/^$parsed_sha1/, split(/\n/, run_git("rev-list --children $_[0]")));
+ if (!@commit_info) {
+ return error_non_fast_forward($_[0]);
+ }
+ my @commit_info_split = split(/ |\n/, $commit_info[0]);
+ # $commit_info_split[1] is the sha1 of the commit to export
+ # $commit_info_split[0] is the sha1 of its direct child
+ push (@commit_pairs, \@commit_info_split);
+ $parsed_sha1 = $commit_info_split[1];
+ }
+ } else {
+ # No remote mediawiki revision. Export the whole
+ # history (linearized with --first-parent)
+ print STDERR "Warning: no common ancestor, pushing complete history\n";
+ my $history = run_git("rev-list --first-parent --children $_[0]");
+ my @history = split('\n', $history);
+ @history = @history[1..$#history];
+ foreach my $line (reverse @history) {
+ my @commit_info_split = split(/ |\n/, $line);
+ push (@commit_pairs, \@commit_info_split);
+ }
+ }
+
+ foreach my $commit_info_split (@commit_pairs) {
+ my $sha1_child = @{$commit_info_split}[0];
+ my $sha1_commit = @{$commit_info_split}[1];
+ my $diff_infos = run_git("diff-tree -r --raw -z $sha1_child $sha1_commit");
+ # TODO: we could detect rename, and encode them with a #redirect on the wiki.
+ # TODO: for now, it's just a delete+add
+ my @diff_info_list = split(/\0/, $diff_infos);
+ # Keep the first line of the commit message as mediawiki comment for the revision
+ my $commit_msg = (split(/\n/, run_git("show --pretty=format:\"%s\" $sha1_commit")))[0];
+ chomp($commit_msg);
+ # Push every blob
+ while (@diff_info_list) {
+ # git diff-tree -z gives an output like
+ # <metadata>\0<filename1>\0
+ # <metadata>\0<filename2>\0
+ # and we've split on \0.
+ my $info = shift(@diff_info_list);
+ my $file = shift(@diff_info_list);
+ mw_push_file($info, $file, $commit_msg);
+ }
+ }
+
+ print STDOUT "ok $_[1]\n";
+ print STDOUT "\n";
+
+ print STDERR "Just pushed some revisions to MediaWiki.\n";
+ print STDERR "The pushed revisions now have to be re-imported, and your current branch\n";
+ print STDERR "needs to be updated with these re-imported commits. You can do this with\n";
+ print STDERR "\n";
+ print STDERR " git pull --rebase\n";
+ print STDERR "\n";
+}
diff --git a/contrib/mw-to-git/git-remote-mediawiki.txt b/contrib/mw-to-git/git-remote-mediawiki.txt
new file mode 100644
index 0000000..4d211f5
--- /dev/null
+++ b/contrib/mw-to-git/git-remote-mediawiki.txt
@@ -0,0 +1,7 @@
+Git-Mediawiki is a project which aims the creation of a gate
+between git and mediawiki, allowing git users to push and pull
+objects from mediawiki just as one would do with a classic git
+repository thanks to remote-helpers.
+
+For more information, visit the wiki at
+https://github.com/Bibzball/Git-Mediawiki/wiki
--
1.7.6.585.g5929f.dirty
^ permalink raw reply related
* [PATCH 1/2] fast-import: initialize variable require_explicit_termination
From: Matthieu Moy @ 2011-08-26 17:11 UTC (permalink / raw)
To: git, gitster; +Cc: Matthieu Moy
The uninitialized variable seems harmless in practice, but let's still be clean.
Signed-off-by: Matthieu Moy <Matthieu.Moy@imag.fr>
---
For some reason, remote helpers seem to be forced to use the "done"
command now. Investing why, I found this, but that wasn't what I was
looking for.
fast-import.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/fast-import.c b/fast-import.c
index 7cc2262..ed8f3cd 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -355,7 +355,7 @@ static unsigned int cmd_save = 100;
static uintmax_t next_mark;
static struct strbuf new_data = STRBUF_INIT;
static int seen_data_command;
-static int require_explicit_termination;
+static int require_explicit_termination = 0;
/* Signal handling */
static volatile sig_atomic_t checkpoint_requested;
--
1.7.6.585.g5929f.dirty
^ permalink raw reply related
* Re: [PATCHv2 2/5] git-tag: introduce long forms for the options
From: Junio C Hamano @ 2011-08-26 17:11 UTC (permalink / raw)
To: Michael J Gruber; +Cc: git, Junio C Hamano, Jeff King
In-Reply-To: <f02e446227a93fff37591f1a866566e6220ce283.1314367414.git.git@drmicha.warpmail.net>
Michael J Gruber <git@drmicha.warpmail.net> writes:
> Long forms are better to memoize, and more reliably uniform across
> commands.
I think people "memorize" and machines "memoize"; machines will do so just
fine without long forms, but I think you are talking about helping people.
The part after ", and" lacks a verb, making it a non-sentence.
> Design notes:
>
> -u,--local-user is named following the analogous gnupg option.
>
> -l,--list is not an argument taking option but a mode switch.
Ok.
The remainder looks good. Thanks.
^ permalink raw reply
* Re: [PATCH 0/5] RFC: patterns for branch list
From: Junio C Hamano @ 2011-08-26 16:55 UTC (permalink / raw)
To: Michael J Gruber; +Cc: Jeff King, git, Michael Schubert
In-Reply-To: <4E5759B1.50705@drmicha.warpmail.net>
Michael J Gruber <git@drmicha.warpmail.net> writes:
> Jeff King venit, vidit, dixit 25.08.2011 19:53:
>> On Thu, Aug 25, 2011 at 10:30:16AM +0200, Michael J Gruber wrote:
>>
>>> Both "tag" and "branch" could activate list mode automatically on an invalid
>>> tag name rather than dieing:
>>>
>>> git tag v1.7.6\*
>>> Warning: tag 'v1.7.6*' not found.
If it is not found, the usual action is create it, no?
>>> v1.7.6
>>> v1.7.6-rc0
>>> v1.7.6-rc1
>>> v1.7.6-rc2
>>> v1.7.6-rc3
>>> v1.7.6.1
>>
>> That just seems confusing to me. What is the exit status? Shouldn't the
>> warning be "error: tag 'v1.7.6*' is not a valid tag name"?
>
> Sure, and sorry, copied the wrong one. I'd just like to have the simple
> way to say "git branch peff/\*" at least as long as we don't have "-l"
> for "--list".
As we use fnmatch() and not match_pathspec() for this pattern matching,
"git branch peff/" will not list all the topics under the peff/ hierarchy
(your example "git branch peff/\*" would be the way), but I would imagine
that we may someday want to update it to allow the leading path match
here. And at that point, distinction between
git branch peff ;# to create a "peff" branch
git branch peff/ ;# to list "peff/" branches, as "peff/" itself is
;# an invalid branch name and your auto listing
;# heuristic kicks in
while it might be very useful for experts, becomes too subtle and would
confuse new people. We should instead require an explicit -l/--list, and
not use the auto listing heuristics (it is fine for -v to imply -l).
^ permalink raw reply
* Re: [PATCHv2] git-replace.txt: Clarify list mode
From: Junio C Hamano @ 2011-08-26 16:30 UTC (permalink / raw)
To: Michael J Gruber; +Cc: git, Christian Couder
In-Reply-To: <0a88518db0b0db8f1a4a4deeebd6dffc2d603e74.1314345131.git.git@drmicha.warpmail.net>
Michael J Gruber <git@drmicha.warpmail.net> writes:
> Clarify that in list mode, "git replace" outputs the shortened ref
> names, not their values.
>
> Also, point to the difficult to find git show-ref $(git replace -l).
>
> Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
> ---
> Documentation/git-replace.txt | 8 ++++++++
> 1 files changed, 8 insertions(+), 0 deletions(-)
>
> diff --git a/Documentation/git-replace.txt b/Documentation/git-replace.txt
> index 17df525..cd00837 100644
> --- a/Documentation/git-replace.txt
> +++ b/Documentation/git-replace.txt
> @@ -61,6 +61,13 @@ OPTIONS
> all if no pattern is given).
> Typing "git replace" without arguments, also lists all replace
> refs.
> ++
> +Note that this lists the names of the replace refs, not their values
> +(not their replacements). You can get the latter like this, e.g.:
Hmm, the update is _not wrong_ per-se, but...
I highly doubt we would want to try to cover confusions that may come from
any and all different mis-/pre-conceptions people may have by making the
description _longer_.
Which part of the wording in the existing description made you think that
the command might list both names and their contents? We should identify
that misleading description (if there is) and fix that, instead of tacking
clarifying clauses at the end.
Given these statements:
"git replace" lists all replace refs.
"ls" lists paths in the directory.
I would say a natural reading of them is that they list "replace refs" and
"paths", not "replace refs and their contents" and "paths and their contents".
By the way, one thing I forgot to say was that I do not think the variant
of the output you wanted to have is necessarily a bad thing (it is bad to
change the existing output to that variant, breaking other
people). Perhaps it can become "-v(erbose)" output?
^ permalink raw reply
* Re: [PATCH 0/2] Add an update=none option for 'loose' submodules
From: Jens Lehmann @ 2011-08-26 16:14 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Heiko Voigt, git
In-Reply-To: <7vbovcg00u.fsf@alter.siamese.dyndns.org>
Am 26.08.2011 08:27, schrieb Junio C Hamano:
> Thinking about it more, I am starting to think that this backwardness may
> be an indication that we are describing a wrong solution to a wrong
> problem.
I Agree, I see two related topics mingled here.
> Isn't the root cause of the issue that a "submodule init" without pathspec
> limit adds everything to .git/config, ending up with all submodules fully
> instantiated, and it is too easy to run such a lazy "submodule init"?
Yes. And that upstream has no option (yet) to tell which submodules should
be automatically initialized on clone, which would get rid of the need to
run "git submodule init" in the first place for the average user.
To me Heiko's change fixes that issue `accidentally` because "update=none"
implies "even when it is initialized I never want to update its work tree,
so don't even clone it". Which is correct but not quite the meaning of the
update config.
> If we allowed the project managers (i.e. the ones who write .gitmodules)
> to specify the default set of submodules to be initialized with such a
> "submodule init", omitting some submodules from even getting registered to
> the recipients' .git/config in the first place, wouldn't that solve the
> issue you are trying to address equally well, without anything to worry
> about this semantic change at all? I am trying to see if we can come up
> with a solution with which we do not even have to add any entry for a
> submodule to .git/config of the superproject, if it is "don't care" kind
> of submodule for the copy of the superproject repository the user has.
I really think we need such an option. I imagine enabling it to even
initialize the submodules on clone or when checking out a branch
introducing a new submodule (and fetch should proactively fetch such
submodules so they can be checked out when disconnected later). Working
title: "init=[auto|no]", with "auto" initializing it on clone and "none"
never initializing it unless the submodule is given on the command line.
> The way in which the project managers specify that a module is not meant
> to be "init"ed by default may be to have "submodule.$name.update = none"
> in the .gitmodules file they ship, so externally there may not be huge
> difference from the behaviour (but not the implementation) of your patch,
> even though submodule.$name.update probably is not a good variable name to
> be used for this purpose.
IMO the update option is about what should happen to an initialized
submodule when a the commit recorded in the superproject changes: Shall it
be checked out, rebased, merged or left alone. Heiko's proposal adds the
missing fourth case, so I think his patch is going in the right direction.
But we should have another config option to state "not to be initialized".
> Another thing we may want to consider is to make .gitmodules describe
> submodule dependencies. If your hypothetical superproject is about a
> library, which consists of doc/, include/, and libsrc/ submodules, with
> pre-built binary perhaps shipped as part of the superproject itself, those
> who work on documentation may want to populate only doc/, those who are
> interested in using the library may want to populate only include/ and
> possibly doc/, and those who work on the library itself would populate
> include/ and libsrc/, possibly with doc/ submodules. It wouldn't make any
> sense to populate libsrc/ without populating include/ submodule, as the
> source would not be buildable without the includes.
Yes, it would be cool having a "git submodule init libsrc" initializing
submodule "include" too.
> Now if we imagine that much more people are interested in using the
> library than working on it, it is plausible that the project may want to
> suggest:
>
> - Majority of people may want to omit libsrc/ submodule; and
>
> - If you populate libsrc/, then you would definitely want to populate
> include/ submodule.
>
> Your "submodule.libsrc.update = none" in .gitmodules can express the
> former, and I think it is natural to express the latter (i.e. submodule
> dependency) in the same file, to be propagated in the same way to the
> consumers.
As I said above, I think "submodule.libsrc.update = none" only does the
trick because "git submodule update" is used to both clone an update a
submodule. I vote for a separate option to express the first condition.
For me this is thread covers three different issues:
1) What submodules should be initialized?
I think we need to add a new option ("init"?) to control that.
2) How should initialized submodules be updated when the SHA1 changes?
This is what "update" should control, and Heiko's patch adds the
missing "none" case.
3) What dependencies do submodules have?
This is slightly orthogonal but should honor the settings in 1) and 2).
And I believe there are use cases where you want the submodule to be
initialized but never updated (and I expect "ignore=all" to be set in
that case too, as the user should not be bothered if the submodule is out
of sync). Imagine large intro movies of a gaming project separated into a
submodule which each developer should have to start the application but
doesn't care if he doesn't have the most up-to-date version.
So before people start (mis-)using the "update=none" option because they
can't use "init=none" yet, I vote for an additional patch implementing
that config option (per submodule and globally). Then we can document each
option properly.
Does that make sense?
^ permalink raw reply
* credential helpers (was: What's cooking in git.git (Aug 2011, #07; Wed, 24))
From: Ted Zlatanov @ 2011-08-26 15:42 UTC (permalink / raw)
To: git
In-Reply-To: <7vhb55i11i.fsf@alter.siamese.dyndns.org>
On Thu, 25 Aug 2011 15:22:49 -0700 Junio C Hamano <gitster@pobox.com> wrote:
JCH> We need to add "auth-domain" support, perhaps from the command line option
JCH> and configuration, if we ever need to support such a site.
JCH> We can consider what you already have as the default case for a more
JCH> general "we cut off at the hostname and take that as the auth-domain
JCH> boundary unless told otherwise". We may not have the way to "tell
JCH> otherwise" yet, but as long as we are reasonably confident that we know
JCH> how to extend the system in a backward compatible way, it is not a
JCH> show-stopper.
How about a config variable with regular expressions like
auth-domain.xyz.url = https://(.*@)?github.com/.*
so then accessing a remote with that URL with or without a username
would pass "auth-domain=xyz" to the helper? If there's no defined
auth-domain then it's not passed to the helper, so it has to just use
the host name (if there is an auth-domain the helper gets it PLUS the
hostname, of course).
I specify the "url" sub-key so we can add more auth-domain selection
criteria or other functionality in the future.
That gives the user a way to do, for instance:
auth-domain.internalcompany.url = https://.*.mycompany.com/.*
I also wanted to suggest that the credential helper should be able to
specify a SSL private user key and the passphrase for it for HTTPS
connections to servers that require such keys.
JCH> The primary reason why I wanted to hold this topic off was because of the
JCH> frequency of bug report we saw this round to topics _after_ they hit the
JCH> "master" branch, indicating that not many people are testing "next" during
JCH> the development cycle as they used to in olden days.
I'll be sure to test credential helpers next week.
Thank you
Ted
^ permalink raw reply
* Re: [RFC/PATCH] attr: map builtin userdiff drivers to well-known extensions
From: Brandon Casey @ 2011-08-26 15:33 UTC (permalink / raw)
To: Jeff King; +Cc: Boaz Harrosh, git
In-Reply-To: <20110826024533.GB17625@sigill.intra.peff.net>
On 08/25/2011 09:45 PM, Jeff King wrote:
> On Thu, Aug 25, 2011 at 05:29:36PM -0500, Brandon Casey wrote:
>
>>> Also, any other extensions that would go into such a list?
>>
>> *.bib diff=bibtex
>> *.tex diff=tex
>
> I had those ones already. ;P
Indeed. I must be blind, I skipped right over them.
>> *.[Ff] diff=fortran
>> *.[Ff][0-9][0-9] diff=fortran
>
> Thanks, I'll add those. I don't see a big problem with generalizing
> f[0-9][0-9] to always be fortran, even though many of those numbers
> aren't used. I don't think I've ever seen one used for anything else.
>
> Should all of our matches be case-insensitive? That is, should we be
> matching both .HTML and .html? Clearly lowercase is the One True Way,
> but I don't know what kind of junk people with case-insensitive
> filesystems have, or whether we should even worry about it.
For the fortran case, Gnu fortran actually processes the files differently
depending on whether the f is capitalized (it preprocesses or not). So
there is a functional reason for using a capital letter.
For the others, I don't know. Do people still create files named .HTML
or is that just a relic of the past? I can't really think of a strong
argument for or against matching insensitively.
-Brandon
^ permalink raw reply
* Re: [PATCH 10/14] http: use hostname in credential description
From: Ted Zlatanov @ 2011-08-26 15:29 UTC (permalink / raw)
To: git
In-Reply-To: <20110825202326.GC6165@sigill.intra.peff.net>
On Thu, 25 Aug 2011 16:23:26 -0400 Jeff King <peff@peff.net> wrote:
JK> On Fri, Aug 19, 2011 at 07:01:21AM -0500, Ted Zlatanov wrote:
>> I see some info in "What's Cooking" about this patch but it's unclear to
>> me whether the hostname issue (where it's hard to have multiple
>> identities on a single server, which I think is all right) is blocking
>> the inclusion of the patch into the next release, or if it will be
>> included eventually if no one complains about that issue, or something
>> else...
JK> Junio and I discussed it a bit in another thread. I think the ability to
JK> use "user@hostname" to disambiguate means the problem is dealt with at a
JK> high level. And the "cache" helper handles that just fine. But the
JK> "store" helper will conflate two entries for the same host. I'll see if
JK> I can work on a patch for that.
Cool, I hope this is the last wrinkle on the bundled helpers.
JK> It looks like Junio is planning to hold the series off until 1.7.8. Have
JK> you been working on a Secrets API helper? If so, I'd love to get
JK> feedback on how well the interface is serving your needs.
Work and Real Life have interfered with coding for fun, but I will have
time next week to try writing a few helpers. This is high priority for
me because of various projects that require it; sorry for taking so long
to start using it.
Ted
^ permalink raw reply
* [PATCH/RFC] submodule: Search for merges only at end of recursive merge
From: Brad King @ 2011-08-26 14:18 UTC (permalink / raw)
To: git; +Cc: gitster, Heiko Voigt
In-Reply-To: <074f22629c034dba738b7241c78229db7f9159ec.1314275112.git.brad.king@kitware.com>
The submodule merge search is not useful during virtual merges because
the results cannot be used automatically. Furthermore any suggesions
made by the search may apply to commits different than HEAD:sub and
MERGE_HEAD:sub, thus confusing the user. Skip searching for submodule
merges during a virtual merge such as that between B and C while merging
the heads of:
B---BC
/ \ /
A X
\ / \
C---CB
Run the search only when the recursion level is zero (!o->call_depth).
This fixes known breakage tested in t7405-submodule-merge.
Signed-off-by: Brad King <brad.king@kitware.com>
---
This addresses the submodule search problem reported in the message
to which this replies. If you think it is correct I can resubmit
both the test and this patch as a single series.
merge-recursive.c | 3 ++-
submodule.c | 11 ++++++++---
submodule.h | 4 +++-
t/t7405-submodule-merge.sh | 2 +-
4 files changed, 14 insertions(+), 6 deletions(-)
diff --git a/merge-recursive.c b/merge-recursive.c
index 0cc1e6f..2118ee9 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -830,7 +830,8 @@ static struct merge_file_info merge_file(struct merge_options *o,
free(result_buf.ptr);
result.clean = (merge_status == 0);
} else if (S_ISGITLINK(a->mode)) {
- result.clean = merge_submodule(result.sha, one->path, one->sha1,
+ result.clean = merge_submodule(o, result.sha,
+ one->path, one->sha1,
a->sha1, b->sha1);
} else if (S_ISLNK(a->mode)) {
hashcpy(result.sha, a->sha1);
diff --git a/submodule.c b/submodule.c
index 1ba9646..c7fdafa 100644
--- a/submodule.c
+++ b/submodule.c
@@ -8,6 +8,7 @@
#include "diffcore.h"
#include "refs.h"
#include "string-list.h"
+#include "merge-recursive.h"
static struct string_list config_name_for_path;
static struct string_list config_fetch_recurse_submodules_for_name;
@@ -642,9 +643,9 @@ static void print_commit(struct commit *commit)
#define MERGE_WARNING(path, msg) \
warning("Failed to merge submodule %s (%s)", path, msg);
-int merge_submodule(unsigned char result[20], const char *path,
- const unsigned char base[20], const unsigned char a[20],
- const unsigned char b[20])
+int merge_submodule(struct merge_options *o, unsigned char result[20],
+ const char *path, const unsigned char base[20],
+ const unsigned char a[20], const unsigned char b[20])
{
struct commit *commit_base, *commit_a, *commit_b;
int parent_count;
@@ -699,6 +700,10 @@ int merge_submodule(unsigned char result[20], const char *path,
* user needs to confirm the resolution.
*/
+ /* This case makes sense only at the main depth 0 merge. */
+ if (o->call_depth)
+ return 0;
+
/* find commit which merges them */
parent_count = find_first_merges(&merges, path, commit_a, commit_b);
switch (parent_count) {
diff --git a/submodule.h b/submodule.h
index 5350b0d..f22172c 100644
--- a/submodule.h
+++ b/submodule.h
@@ -2,6 +2,7 @@
#define SUBMODULE_H
struct diff_options;
+struct merge_options;
enum {
RECURSE_SUBMODULES_ON_DEMAND = -1,
@@ -27,7 +28,8 @@ int fetch_populated_submodules(int num_options, const char **options,
const char *prefix, int command_line_option,
int quiet);
unsigned is_submodule_modified(const char *path, int ignore_untracked);
-int merge_submodule(unsigned char result[20], const char *path, const unsigned char base[20],
+int merge_submodule(struct merge_options *o, unsigned char result[20],
+ const char *path, const unsigned char base[20],
const unsigned char a[20], const unsigned char b[20]);
#endif
diff --git a/t/t7405-submodule-merge.sh b/t/t7405-submodule-merge.sh
index 14da2e3..0d5b42a 100755
--- a/t/t7405-submodule-merge.sh
+++ b/t/t7405-submodule-merge.sh
@@ -269,7 +269,7 @@ test_expect_success 'setup for recursive merge with submodule' '
'
# merge should leave submodule unmerged in index
-test_expect_failure 'recursive merge with submodule' '
+test_expect_success 'recursive merge with submodule' '
(cd merge-recursive &&
test_must_fail git merge top-bc &&
echo "160000 $(git rev-parse top-cb:sub) 2 sub" > expect2 &&
--
1.7.4.4
^ permalink raw reply related
* Re: Files that cannot be added to the index
From: Michael J Gruber @ 2011-08-26 14:10 UTC (permalink / raw)
To: seanh; +Cc: git
In-Reply-To: <CAMvu5bLuRWinMYNc4NoRKQKiLCWLcwkpowEFT4GQ0mcJYj6eOg@mail.gmail.com>
seanh venit, vidit, dixit 26.08.2011 14:26:
> Can anyone guess what's going on when I have a modified file that
> shows up in `git status`, but the file cannot be added to the index
> (or committed)? `git add FILE` does nothing, the file still shows as
> modified but not added in `git status`.
>
> I have two different repos that have each developed this problem with
> two different files. I don't know how it happened. The problem occurs
> wherever the repos are cloned. Even if I delete the local copy (where
> I'm seeing the problem) and clone the repo again from elsewhere,
> problem persists.
A log of your commands (or access to the repo) would help diagnose that,
along with information about the system and the git version.
Do you "clone" by making a copy, by any chance?
Michael
^ permalink raw reply
* [PATCHv2 5/5] branch: allow pattern arguments
From: Michael J Gruber @ 2011-08-26 14:05 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Jeff King
In-Reply-To: <cover.1314367414.git.git@drmicha.warpmail.net>
Allow pattern arguments for the list mode just like for git tag -l.
Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
---
Documentation/git-branch.txt | 8 ++++++--
builtin/branch.c | 24 +++++++++++++++++++++---
t/t3203-branch-output.sh | 10 ++++++++++
3 files changed, 37 insertions(+), 5 deletions(-)
diff --git a/Documentation/git-branch.txt b/Documentation/git-branch.txt
index ac278fb..2b8bc84 100644
--- a/Documentation/git-branch.txt
+++ b/Documentation/git-branch.txt
@@ -10,7 +10,7 @@ SYNOPSIS
[verse]
'git branch' [--color[=<when>] | --no-color] [-r | -a]
[--list] [-v [--abbrev=<length> | --no-abbrev]]
- [(--merged | --no-merged | --contains) [<commit>]]
+ [(--merged | --no-merged | --contains) [<commit>]] [<pattern>...]
'git branch' [--set-upstream | --track | --no-track] [-l] [-f] <branchname> [<start-point>]
'git branch' (-m | -M) [<oldbranch>] <newbranch>
'git branch' (-d | -D) [-r] <branchname>...
@@ -22,6 +22,9 @@ With no arguments, existing branches are listed and the current branch will
be highlighted with an asterisk. Option `-r` causes the remote-tracking
branches to be listed, and option `-a` shows both. This list mode is also
activated by the `--list` and `-v` options (see below).
+<pattern> restricts the output to matching branches, the pattern is a shell
+wildcard (i.e., matched using fnmatch(3))
+Multiple patterns may be given; if any of them matches, the tag is shown.
With `--contains`, shows only the branches that contain the named commit
(in other words, the branches whose tip commits are descendants of the
@@ -112,7 +115,8 @@ OPTIONS
List both remote-tracking branches and local branches.
--list::
- Activate the list mode.
+ Activate the list mode. `git branch <pattern>` would try to create a branch,
+ use `git branch --list <pattern>` to list matching branches.
-v::
--verbose::
diff --git a/builtin/branch.c b/builtin/branch.c
index 4a33b07..e6bef49 100644
--- a/builtin/branch.c
+++ b/builtin/branch.c
@@ -260,9 +260,22 @@ static char *resolve_symref(const char *src, const char *prefix)
struct append_ref_cb {
struct ref_list *ref_list;
+ const char **pattern;
int ret;
};
+static int match_patterns(const char **pattern, const char *refname)
+{
+ if (!*pattern)
+ return 1; /* no pattern always matches */
+ while (*pattern) {
+ if (!fnmatch(*pattern, refname, 0))
+ return 1;
+ pattern++;
+ }
+ return 0;
+}
+
static int append_ref(const char *refname, const unsigned char *sha1, int flags, void *cb_data)
{
struct append_ref_cb *cb = (struct append_ref_cb *)(cb_data);
@@ -297,6 +310,9 @@ static int append_ref(const char *refname, const unsigned char *sha1, int flags,
if ((kind & ref_list->kinds) == 0)
return 0;
+ if (!match_patterns(cb->pattern, refname))
+ return 0;
+
commit = NULL;
if (ref_list->verbose || ref_list->with_commit || merge_filter != NO_FILTER) {
commit = lookup_commit_reference_gently(sha1, 1);
@@ -492,7 +508,7 @@ static void show_detached(struct ref_list *ref_list)
}
}
-static int print_ref_list(int kinds, int detached, int verbose, int abbrev, struct commit_list *with_commit)
+static int print_ref_list(int kinds, int detached, int verbose, int abbrev, struct commit_list *with_commit, const char **pattern)
{
int i;
struct append_ref_cb cb;
@@ -506,6 +522,7 @@ static int print_ref_list(int kinds, int detached, int verbose, int abbrev, stru
if (merge_filter != NO_FILTER)
init_revisions(&ref_list.revs, NULL);
cb.ref_list = &ref_list;
+ cb.pattern = pattern;
cb.ret = 0;
for_each_rawref(append_ref, &cb);
if (merge_filter != NO_FILTER) {
@@ -523,7 +540,7 @@ static int print_ref_list(int kinds, int detached, int verbose, int abbrev, stru
qsort(ref_list.list, ref_list.index, sizeof(struct ref_item), ref_cmp);
detached = (detached && (kinds & REF_LOCAL_BRANCH));
- if (detached)
+ if (detached && match_patterns(pattern, "HEAD"))
show_detached(&ref_list);
for (i = 0; i < ref_list.index; i++) {
@@ -701,7 +718,8 @@ int cmd_branch(int argc, const char **argv, const char *prefix)
if (delete)
return delete_branches(argc, argv, delete > 1, kinds);
else if (list)
- return print_ref_list(kinds, detached, verbose, abbrev, with_commit);
+ return print_ref_list(kinds, detached, verbose, abbrev,
+ with_commit, argv);
else if (rename && (argc == 1))
rename_branch(head, argv[0], rename > 1);
else if (rename && (argc == 2))
diff --git a/t/t3203-branch-output.sh b/t/t3203-branch-output.sh
index 61e095c..f2b294b 100755
--- a/t/t3203-branch-output.sh
+++ b/t/t3203-branch-output.sh
@@ -81,6 +81,16 @@ test_expect_success 'git branch -v shows branch summaries' '
'
cat >expect <<'EOF'
+two
+one
+EOF
+test_expect_success 'git branch -v pattern shows branch summaries' '
+ git branch -v branch* >tmp &&
+ awk "{print \$NF}" <tmp >actual &&
+ test_cmp expect actual
+'
+
+cat >expect <<'EOF'
* (no branch)
branch-one
branch-two
--
1.7.6.845.gc3c05
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox